diff --git a/README.md b/README.md index fd56098..d2fd3be 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ Try it without installing anything: [live demo](https://huggingface.co/spaces/Un | Sliding-window long documents + streaming scan | **Included** | -On the neuralchemy prompt-injection set, regex-only detection reaches **F1 0.58 / recall 0.41** — a fast first line, not sufficient alone. Adding the ML span model (`Guard(model="tiny")`) takes that to **F1 0.99 / recall 0.98**, and lifts recall on *indirect* injection from **0.05 → 0.91**. False-positive rate stays under 1% on the injection set (2.1% on a separate hard-benign corpus). Full tables, methodology, and honest caveats: [`sdk/docs/BENCHMARKS.md`](sdk/docs/BENCHMARKS.md). Per-axis model metrics (including failure modes) are on the [model card](https://huggingface.co/Unplug-AI/unplug-tiny-v1). +On the held-out `core/test` split of the neuralchemy prompt-injection set (942 rows), regex-only detection reaches **F1 0.52 / recall 0.35**, a fast first line and not sufficient alone. Adding the ML span model (`Guard(model="tiny")`) takes that to **F1 0.97 / recall 0.96**, and lifts recall on *indirect* injection from **0.05 to 0.91**. False positives on that split run at 5 of 390 benign rows, and 2 of 95 on a separate hand-written benign corpus. On broader public benign sets the model over-flags badly, up to 34% of a combined 3,227-prompt validation set, so tune the threshold for your own traffic rather than trusting the defaults. Full tables, methodology, and the axes where it fails: [`sdk/docs/BENCHMARKS.md`](sdk/docs/BENCHMARKS.md). Per-axis model metrics are on the [model card](https://huggingface.co/Unplug-AI/unplug-tiny-v1). **Language support.** Regex + normalization detection is tuned for English today. Ordinary non-English input (Cyrillic, CJK, etc.) is *not* treated as an evasion diff --git a/sdk/README.md b/sdk/README.md index 36172fc..234f8da 100644 --- a/sdk/README.md +++ b/sdk/README.md @@ -73,15 +73,21 @@ single-turn sessions: | Dataset | Mode | Recall | F1 | FPR | | --- | --- | ---: | ---: | ---: | -| neuralchemy (direct, 4,391) | regex-only | 0.41 | 0.58 | <1% | -| neuralchemy (direct, 4,391) | **regex + ML** | **0.98** | **0.99** | <1% | -| microsoft llmail (indirect, 2,500) | regex-only | 0.05 | — | — | -| microsoft llmail (indirect, 2,500) | **regex + ML** | **0.91** | — | — | - -Precision stays ~0.99 in both modes. The `<1%` FPR is on the injection set; on a -separate hard-benign corpus (95 prompts) regex flags 0 and regex + ML flags 2, -i.e. **2.1%** (one of which is a *review*, not a block). `inj_threshold` is tuned -to the recall/FPR knee. +| neuralchemy `core/test` (direct, 942) | regex-only | 0.35 | 0.52 | 1/390 | +| neuralchemy `core/test` (direct, 942) | **regex + ML** | **0.96** | **0.97** | 5/390 | +| microsoft llmail (indirect, 2,500) | regex-only | 0.05 | n/a | n/a | +| microsoft llmail (indirect, 2,500) | **regex + ML** | **0.91** | n/a | n/a | + +`core/test` is the held-out split. `unplug-tiny` was fine-tuned on `core/train`, so +numbers measured there are memorisation rather than detection. Precision stays near +0.99 in both modes. The FPR column counts benign rows inside the injection set; on a +separate hand-written benign corpus of 95 prompts, regex flags 0 and regex + ML flags 2, +one of which is a *review* rather than a block. `inj_threshold` is tuned to the +recall/FPR knee. + +On broader public benign sets the model over-flags badly, up to 34% of a combined +3,227-prompt validation set. See [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) before +trusting the defaults. **Language support.** Regex + normalization detection is tuned for English today. Ordinary non-English input (Cyrillic, CJK, etc.) is *not* treated as an evasion diff --git a/sdk/benchmarks/download.py b/sdk/benchmarks/download.py index 8ec4140..c43d8a3 100644 --- a/sdk/benchmarks/download.py +++ b/sdk/benchmarks/download.py @@ -26,10 +26,18 @@ def _export_jsonl(samples: list[Sample], path: Path) -> None: f.write(json.dumps(row) + "\n") -def download_neuralchemy(out_dir: Path, *, limit: int | None = None) -> Path: +# unplug-tiny was fine-tuned on the "train" split of this dataset, so scoring the +# model against it measures memorisation. "test" is the held-out split the model +# card gates on. Do not change this default. +NEURALCHEMY_SPLIT = "test" + + +def download_neuralchemy( + out_dir: Path, *, limit: int | None = None, split: str = NEURALCHEMY_SPLIT +) -> Path: from datasets import load_dataset - ds = load_dataset("neuralchemy/Prompt-injection-dataset", split="train") + ds = load_dataset("neuralchemy/Prompt-injection-dataset", "core", split=split) samples: list[Sample] = [] for i, row in enumerate(ds): if limit is not None and i >= limit: @@ -90,12 +98,18 @@ def main() -> None: ) parser.add_argument("--out", type=Path, default=None) parser.add_argument("--limit", type=int, default=None) + parser.add_argument( + "--neuralchemy-split", + default=NEURALCHEMY_SPLIT, + choices=["train", "validation", "test"], + help="train overlaps unplug-tiny fine-tuning data; results are not comparable", + ) args = parser.parse_args() out_dir = args.out or _repo_datasets_dir() paths: list[Path] = [] if args.dataset in ("neuralchemy", "all"): - paths.append(download_neuralchemy(out_dir, limit=args.limit)) + paths.append(download_neuralchemy(out_dir, limit=args.limit, split=args.neuralchemy_split)) if args.dataset in ("microsoft", "all"): paths.append(download_microsoft_subset(out_dir, limit=args.limit or 5000)) for p in paths: diff --git a/sdk/docs/BENCHMARKS.md b/sdk/docs/BENCHMARKS.md index 7f98326..8bc62f5 100644 --- a/sdk/docs/BENCHMARKS.md +++ b/sdk/docs/BENCHMARKS.md @@ -1,9 +1,12 @@ # SDK benchmark results -- **Date:** 2026-06-15 (ML rows); regex-only neuralchemy refreshed **2026-07-20** (Phase C) +- **Date:** 2026-08-25 - **Guard:** `unplug-ai`, default scanners - **Model:** `unplug-tiny-v1` (DeBERTa-v3-xsmall dual-head span model), `Guard(model="tiny")` -- **Detection threshold:** risk ≥ 0.5 counts as flagged (block or review) +- **Detection threshold:** risk >= 0.5 counts as flagged (block or review) +- **Split:** `neuralchemy/Prompt-injection-dataset` `core/test` (942 rows). `unplug-tiny` + was fine-tuned on `core/train`, so any score measured there is memorisation, not + detection. Earlier revisions of this page reported the train split by mistake. - **Methodology:** isolated single-turn sessions. Each sample is scanned in a fresh `ExecutionContext` (`scan_request(..., isolated=True)`), so multi-turn trajectory state never leaks between independent samples. @@ -17,13 +20,28 @@ the regex layer alone misses (especially **indirect** injection). | Dataset | Samples | Mode | F1 | Recall | FPR | Precision | | --- | ---: | --- | ---: | ---: | ---: | ---: | -| neuralchemy/Prompt-injection-dataset | 4,391 | regex-only | 0.575 | 0.405 | 0.0052 | 0.992 | -| neuralchemy/Prompt-injection-dataset | 4,391 | **regex + ML** | **0.987** | **0.981** | 0.0098 | 0.994 | -| microsoft/llmail-inject (Phase1 subset, attacks) | 2,500 | regex-only | — | 0.052 | — | — | -| microsoft/llmail-inject (Phase1 subset, attacks) | 2,500 | **regex + ML** | — | **0.907** | — | — | +| neuralchemy `core/test` | 942 | regex-only | 0.519 | 0.351 | 0.0026 | 0.995 | +| neuralchemy `core/test` | 942 | **regex + ML** | **0.974** | **0.958** | 0.0128 | 0.991 | +| microsoft/llmail-inject (Phase1 subset, attacks) | 2,500 | regex-only | n/a | 0.052 | n/a | n/a | +| microsoft/llmail-inject (Phase1 subset, attacks) | 2,500 | **regex + ML** | n/a | **0.907** | n/a | n/a | -- **Direct injection (neuralchemy):** recall **0.41 → 0.98**, F1 **0.58 → 0.99**, precision ~0.99 in both modes, false-positive rate stays under 1%. -- **Indirect injection (microsoft):** recall **0.05 → 0.91**. Regex is structurally blind to indirect injection; the ML pass is what makes it detectable. +- **Direct injection (neuralchemy):** recall **0.35 -> 0.96**, F1 **0.52 -> 0.97**. + Precision holds near 0.99 in both modes. The ML pass costs false positives: + FPR goes from 1 false positive in 390 benign rows to 5. +- **Indirect injection (microsoft):** recall **0.05 -> 0.91**. Regex is structurally + blind to indirect injection; the ML pass is what makes it detectable. We have not + confirmed whether any of this subset overlaps `unplug-tiny` fine-tuning data, so + read that row as an upper bound. + +## What changed on 2026-08-25 + +This page used to report `core/train`: 0.987 F1 and 0.981 recall for regex + ML. +That split is the model's own fine-tuning data. On the held-out `core/test` split the +same build scores 0.974 F1 and 0.958 recall, so contamination was worth about 2.3 +points of recall. + +Regex-only moved too, 0.405 recall down to 0.351, and regex has no training data at +all. Part of the gap is that the two splits are not equally hard, not memorisation. ## False-positive rate on clean traffic @@ -59,6 +77,28 @@ recall/FPR knee) to keep this rate low without sacrificing recall. - The microsoft subset is attacks-only (recall, no FPR). neuralchemy carries both labels. +## Where it does badly + +Detection numbers on corpora we did not pick are worse, and the false-positive rate on +broad benign traffic is much worse. From the `unplug-tiny-v1` model card, which runs a +frozen harness over public sets: + +| Set | Recall | Doc FPR | F1 | +| --- | ---: | ---: | ---: | +| BIPIA indirect proxy (1,242) | 0.973 | 0.000 | 0.986 | +| InjecGuard validation (144) | 0.896 | 0.208 | 0.775 | +| Deepset full (662) | 0.829 | 0.188 | 0.784 | +| spikee contextual (986) | 0.786 | 0.067 | 0.879 | +| LLM-PIEval agentic (750) | 0.761 | n/a | 0.865 | +| OOD direct injection (281) | 0.619 | 0.102 | 0.692 | +| WildGuard benign (971) | n/a | 0.542 | n/a | +| Combined public validation (3,227) | 0.810 | 0.341 | 0.717 | + +One third of benign prompts in the combined public set get flagged. If your traffic +looks like WildGuard rather than like our benign corpus, expect over-blocking, and +tune the threshold before you put this in front of users. The model card carries the +per-axis failure modes and marks three of its own gates as failing. + ## Reproduce ```bash diff --git a/sdk/docs/EVAL_PHASE_C.md b/sdk/docs/EVAL_PHASE_C.md index 790c74b..22a4bdd 100644 --- a/sdk/docs/EVAL_PHASE_C.md +++ b/sdk/docs/EVAL_PHASE_C.md @@ -13,7 +13,7 @@ uv sync --all-extras --dev # Datasets are already under benchmarks/data/. To refresh from Hugging Face: uv run python -m benchmarks.download --dataset all --out benchmarks/data -# neuralchemy: full train export; microsoft: streaming Phase1 subset (default --limit 5000) +# neuralchemy: core/test holdout; microsoft: streaming Phase1 subset (default --limit 5000) # Smoke (built-in samples) uv run python -c " @@ -30,6 +30,10 @@ uv run python -m benchmarks.run benchmarks/data/microsoft_indirect.jsonl --isola uv run python -m benchmarks.run benchmarks/data/neuralchemy.jsonl --ml --isolated --format json ``` +> **Superseded.** The neuralchemy rows below were measured on `core/train`, which is +> `unplug-tiny`'s fine-tuning data. They are kept as a record of what that run reported. +> Current numbers are on the `core/test` holdout in [`BENCHMARKS.md`](BENCHMARKS.md). + ## Results (this run) | Dataset | Samples | Mode | Precision | Recall | F1 | FPR |