bench: evaluate on the held-out split, not the training data

README.md

@@ -75,7 +75,7 @@ Try it without installing anything: [live demo](https://huggingface.co/spaces/Un
 | Sliding-window long documents + streaming scan | **Included** |
 
 
-On the neuralchemy prompt-injection set, regex-only detection reaches **F1 0.58 / recall 0.41** — a fast first line, not sufficient alone. Adding the ML span model (`Guard(model="tiny")`) takes that to **F1 0.99 / recall 0.98**, and lifts recall on *indirect* injection from **0.05 → 0.91**. False-positive rate stays under 1% on the injection set (2.1% on a separate hard-benign corpus). Full tables, methodology, and honest caveats: [`sdk/docs/BENCHMARKS.md`](sdk/docs/BENCHMARKS.md). Per-axis model metrics (including failure modes) are on the [model card](https://huggingface.co/Unplug-AI/unplug-tiny-v1).
+On the held-out `core/test` split of the neuralchemy prompt-injection set (942 rows), regex-only detection reaches **F1 0.52 / recall 0.35**, a fast first line and not sufficient alone. Adding the ML span model (`Guard(model="tiny")`) takes that to **F1 0.97 / recall 0.96**, and lifts recall on *indirect* injection from **0.05 to 0.91**. False positives on that split run at 5 of 390 benign rows, and 2 of 95 on a separate hand-written benign corpus. On broader public benign sets the model over-flags badly, up to 34% of a combined 3,227-prompt validation set, so tune the threshold for your own traffic rather than trusting the defaults. Full tables, methodology, and the axes where it fails: [`sdk/docs/BENCHMARKS.md`](sdk/docs/BENCHMARKS.md). Per-axis model metrics are on the [model card](https://huggingface.co/Unplug-AI/unplug-tiny-v1).
 
 **Language support.** Regex + normalization detection is tuned for English today.
 Ordinary non-English input (Cyrillic, CJK, etc.) is *not* treated as an evasion

sdk/README.md

@@ -73,15 +73,21 @@ single-turn sessions:
 
 | Dataset | Mode | Recall | F1 | FPR |
 | --- | --- | ---: | ---: | ---: |
-| neuralchemy (direct, 4,391) | regex-only | 0.41 | 0.58 | <1% |
-| neuralchemy (direct, 4,391) | **regex + ML** | **0.98** | **0.99** | <1% |
-| microsoft llmail (indirect, 2,500) | regex-only | 0.05 | — | — |
-| microsoft llmail (indirect, 2,500) | **regex + ML** | **0.91** | — | — |
-
-Precision stays ~0.99 in both modes. The `<1%` FPR is on the injection set; on a
-separate hard-benign corpus (95 prompts) regex flags 0 and regex + ML flags 2,
-i.e. **2.1%** (one of which is a *review*, not a block). `inj_threshold` is tuned
-to the recall/FPR knee.
+| neuralchemy `core/test` (direct, 942) | regex-only | 0.35 | 0.52 | 1/390 |
+| neuralchemy `core/test` (direct, 942) | **regex + ML** | **0.96** | **0.97** | 5/390 |
+| microsoft llmail (indirect, 2,500) | regex-only | 0.05 | n/a | n/a |
+| microsoft llmail (indirect, 2,500) | **regex + ML** | **0.91** | n/a | n/a |
+
+`core/test` is the held-out split. `unplug-tiny` was fine-tuned on `core/train`, so
+numbers measured there are memorisation rather than detection. Precision stays near
+0.99 in both modes. The FPR column counts benign rows inside the injection set; on a
+separate hand-written benign corpus of 95 prompts, regex flags 0 and regex + ML flags 2,
+one of which is a *review* rather than a block. `inj_threshold` is tuned to the
+recall/FPR knee.
+
+On broader public benign sets the model over-flags badly, up to 34% of a combined
+3,227-prompt validation set. See [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md) before
+trusting the defaults.
 
 **Language support.** Regex + normalization detection is tuned for English today.
 Ordinary non-English input (Cyrillic, CJK, etc.) is *not* treated as an evasion

sdk/benchmarks/download.py

@@ -26,10 +26,18 @@ def _export_jsonl(samples: list[Sample], path: Path) -> None:
             f.write(json.dumps(row) + "\n")
 
 
-def download_neuralchemy(out_dir: Path, *, limit: int | None = None) -> Path:
+# unplug-tiny was fine-tuned on the "train" split of this dataset, so scoring the
+# model against it measures memorisation. "test" is the held-out split the model
+# card gates on. Do not change this default.
+NEURALCHEMY_SPLIT = "test"
+
+
+def download_neuralchemy(
+    out_dir: Path, *, limit: int | None = None, split: str = NEURALCHEMY_SPLIT
+) -> Path:
     from datasets import load_dataset
 
-    ds = load_dataset("neuralchemy/Prompt-injection-dataset", split="train")
+    ds = load_dataset("neuralchemy/Prompt-injection-dataset", "core", split=split)
     samples: list[Sample] = []
     for i, row in enumerate(ds):
         if limit is not None and i >= limit:
@@ -90,12 +98,18 @@ def main() -> None:
     )
     parser.add_argument("--out", type=Path, default=None)
     parser.add_argument("--limit", type=int, default=None)
+    parser.add_argument(
+        "--neuralchemy-split",
+        default=NEURALCHEMY_SPLIT,
+        choices=["train", "validation", "test"],
+        help="train overlaps unplug-tiny fine-tuning data; results are not comparable",
+    )
     args = parser.parse_args()
 
     out_dir = args.out or _repo_datasets_dir()
     paths: list[Path] = []
     if args.dataset in ("neuralchemy", "all"):
-        paths.append(download_neuralchemy(out_dir, limit=args.limit))
+        paths.append(download_neuralchemy(out_dir, limit=args.limit, split=args.neuralchemy_split))
     if args.dataset in ("microsoft", "all"):
         paths.append(download_microsoft_subset(out_dir, limit=args.limit or 5000))
     for p in paths:

sdk/docs/BENCHMARKS.md

@@ -1,9 +1,12 @@
 # SDK benchmark results
 
-- **Date:** 2026-06-15 (ML rows); regex-only neuralchemy refreshed **2026-07-20** (Phase C)
+- **Date:** 2026-08-25
 - **Guard:** `unplug-ai`, default scanners
 - **Model:** `unplug-tiny-v1` (DeBERTa-v3-xsmall dual-head span model), `Guard(model="tiny")`
-- **Detection threshold:** risk ≥ 0.5 counts as flagged (block or review)
+- **Detection threshold:** risk >= 0.5 counts as flagged (block or review)
+- **Split:** `neuralchemy/Prompt-injection-dataset` `core/test` (942 rows). `unplug-tiny`
+  was fine-tuned on `core/train`, so any score measured there is memorisation, not
+  detection. Earlier revisions of this page reported the train split by mistake.
 - **Methodology:** isolated single-turn sessions. Each sample is scanned in a fresh
   `ExecutionContext` (`scan_request(..., isolated=True)`), so multi-turn trajectory
   state never leaks between independent samples.
@@ -17,13 +20,28 @@ the regex layer alone misses (especially **indirect** injection).
 
 | Dataset | Samples | Mode | F1 | Recall | FPR | Precision |
 | --- | ---: | --- | ---: | ---: | ---: | ---: |
-| neuralchemy/Prompt-injection-dataset | 4,391 | regex-only | 0.575 | 0.405 | 0.0052 | 0.992 |
-| neuralchemy/Prompt-injection-dataset | 4,391 | **regex + ML** | **0.987** | **0.981** | 0.0098 | 0.994 |
-| microsoft/llmail-inject (Phase1 subset, attacks) | 2,500 | regex-only | — | 0.052 | — | — |
-| microsoft/llmail-inject (Phase1 subset, attacks) | 2,500 | **regex + ML** | — | **0.907** | — | — |
+| neuralchemy `core/test` | 942 | regex-only | 0.519 | 0.351 | 0.0026 | 0.995 |
+| neuralchemy `core/test` | 942 | **regex + ML** | **0.974** | **0.958** | 0.0128 | 0.991 |
+| microsoft/llmail-inject (Phase1 subset, attacks) | 2,500 | regex-only | n/a | 0.052 | n/a | n/a |
+| microsoft/llmail-inject (Phase1 subset, attacks) | 2,500 | **regex + ML** | n/a | **0.907** | n/a | n/a |
 
-- **Direct injection (neuralchemy):** recall **0.41 → 0.98**, F1 **0.58 → 0.99**, precision ~0.99 in both modes, false-positive rate stays under 1%.
-- **Indirect injection (microsoft):** recall **0.05 → 0.91**. Regex is structurally blind to indirect injection; the ML pass is what makes it detectable.
+- **Direct injection (neuralchemy):** recall **0.35 -> 0.96**, F1 **0.52 -> 0.97**.
+  Precision holds near 0.99 in both modes. The ML pass costs false positives:
+  FPR goes from 1 false positive in 390 benign rows to 5.
+- **Indirect injection (microsoft):** recall **0.05 -> 0.91**. Regex is structurally
+  blind to indirect injection; the ML pass is what makes it detectable. We have not
+  confirmed whether any of this subset overlaps `unplug-tiny` fine-tuning data, so
+  read that row as an upper bound.
+
+## What changed on 2026-08-25
+
+This page used to report `core/train`: 0.987 F1 and 0.981 recall for regex + ML.
+That split is the model's own fine-tuning data. On the held-out `core/test` split the
+same build scores 0.974 F1 and 0.958 recall, so contamination was worth about 2.3
+points of recall.
+
+Regex-only moved too, 0.405 recall down to 0.351, and regex has no training data at
+all. Part of the gap is that the two splits are not equally hard, not memorisation.
 
 ## False-positive rate on clean traffic
 
@@ -59,6 +77,28 @@ recall/FPR knee) to keep this rate low without sacrificing recall.
 - The microsoft subset is attacks-only (recall, no FPR). neuralchemy carries both
   labels.
 
+## Where it does badly
+
+Detection numbers on corpora we did not pick are worse, and the false-positive rate on
+broad benign traffic is much worse. From the `unplug-tiny-v1` model card, which runs a
+frozen harness over public sets:
+
+| Set | Recall | Doc FPR | F1 |
+| --- | ---: | ---: | ---: |
+| BIPIA indirect proxy (1,242) | 0.973 | 0.000 | 0.986 |
+| InjecGuard validation (144) | 0.896 | 0.208 | 0.775 |
+| Deepset full (662) | 0.829 | 0.188 | 0.784 |
+| spikee contextual (986) | 0.786 | 0.067 | 0.879 |
+| LLM-PIEval agentic (750) | 0.761 | n/a | 0.865 |
+| OOD direct injection (281) | 0.619 | 0.102 | 0.692 |
+| WildGuard benign (971) | n/a | 0.542 | n/a |
+| Combined public validation (3,227) | 0.810 | 0.341 | 0.717 |
+
+One third of benign prompts in the combined public set get flagged. If your traffic
+looks like WildGuard rather than like our benign corpus, expect over-blocking, and
+tune the threshold before you put this in front of users. The model card carries the
+per-axis failure modes and marks three of its own gates as failing.
+
 ## Reproduce
 
 ```bash

sdk/docs/EVAL_PHASE_C.md

@@ -13,7 +13,7 @@ uv sync --all-extras --dev
 
 # Datasets are already under benchmarks/data/. To refresh from Hugging Face:
 uv run python -m benchmarks.download --dataset all --out benchmarks/data
-# neuralchemy: full train export; microsoft: streaming Phase1 subset (default --limit 5000)
+# neuralchemy: core/test holdout; microsoft: streaming Phase1 subset (default --limit 5000)
 
 # Smoke (built-in samples)
 uv run python -c "
@@ -30,6 +30,10 @@ uv run python -m benchmarks.run benchmarks/data/microsoft_indirect.jsonl --isola
 uv run python -m benchmarks.run benchmarks/data/neuralchemy.jsonl --ml --isolated --format json
 ```
 
+> **Superseded.** The neuralchemy rows below were measured on `core/train`, which is
+> `unplug-tiny`'s fine-tuning data. They are kept as a record of what that run reported.
+> Current numbers are on the `core/test` holdout in [`BENCHMARKS.md`](BENCHMARKS.md).
+
 ## Results (this run)
 
 | Dataset | Samples | Mode | Precision | Recall | F1 | FPR |