release: 4.8.0

.codecov.yml

@@ -1 +1,10 @@
 comment: no
+
+codecov:
+  notify:
+    # The CI test matrix uploads 12 coverage reports (4 Python versions x 3
+    # install profiles). Only the nlp-advanced profile installs litellm, so
+    # the guardrail adapter's lines look uncovered until those uploads land.
+    # Waiting for all builds prevents the recurring interim-red patch status.
+    after_n_builds: 12
+    wait_for_ci: true

.gitignore

@@ -69,5 +69,4 @@ docs/*
 */**/__pycache__/
 
 notes/benchmarking_notes.md
-Roadmap.md
 notes/*

AGENTS.md

@@ -6,7 +6,7 @@
 
 ## Core Value Proposition
 
-- **Ultra-Fast Performance**: 190x faster than spaCy for structured PII, 32x faster with GLiNER
+- **Ultra-Fast Performance**: 100x+ faster than spaCy/Presidio for structured PII (reproduce: `python benchmarks/run.py`), 32x faster with GLiNER
 - **Lightweight Core**: <2MB package with optional ML extras
 - **Modern Engine Options**: Regex, GLiNER, spaCy, and smart cascading
 - **Production Ready**: Comprehensive testing, CI/CD, and performance validation
@@ -24,7 +24,7 @@
 - **GLiNER Integration**: Modern NER engine with PII-specialized models
 - **Smart Cascading**: Intelligent regex → GLiNER → spaCy progression
 - **Enhanced CLI**: Model management with `--engine` flags
-- **Performance Validation**: 190x regex, 32x GLiNER benchmarks confirmed
+- **Performance Validation**: 100x+ regex speedup reproducible via `benchmarks/`; 32x GLiNER
 - **CI/CD Consolidation**: 7 workflows → 3 (ci, release, benchmark)
 
 ## Quick Development Setup
@@ -56,7 +56,7 @@ python -c "from datafog.services.text_service import TextService; print('✅ All
 from datafog.services.text_service import TextService
 
 # Core engines (always available)
-regex_service = TextService(engine="regex")      # 190x faster, structured PII
+regex_service = TextService(engine="regex")      # 100x+ faster, structured PII
 
 # ML engines (require extras)
 gliner_service = TextService(engine="gliner")    # 32x faster, modern NER
@@ -69,12 +69,12 @@ auto_service = TextService(engine="auto")        # Legacy: regex→spaCy
 
 ### Performance Comparison (Validated)
 
-| Engine   | Speed vs spaCy  | Accuracy          | Use Case                    | Install          |
-| -------- | --------------- | ----------------- | --------------------------- | ---------------- |
-| `regex`  | **190x faster** | High (structured) | Emails, phones, SSNs        | Core only        |
-| `gliner` | **32x faster**  | Very High         | Modern NER, custom entities | `[nlp-advanced]` |
-| `spacy`  | 1x (baseline)   | Good              | Traditional NLP             | `[nlp]`          |
-| `smart`  | **60x faster**  | Highest           | Best balance                | `[nlp-advanced]` |
+| Engine   | Speed vs spaCy   | Accuracy          | Use Case                    | Install          |
+| -------- | ---------------- | ----------------- | --------------------------- | ---------------- |
+| `regex`  | **100x+ faster** | High (structured) | Emails, phones, SSNs        | Core only        |
+| `gliner` | **32x faster**   | Very High         | Modern NER, custom entities | `[nlp-advanced]` |
+| `spacy`  | 1x (baseline)    | Good              | Traditional NLP             | `[nlp]`          |
+| `smart`  | **60x faster**   | Highest           | Best balance                | `[nlp-advanced]` |
 
 ### Dependency Strategy
 
@@ -281,7 +281,7 @@ export PYTHONPATH=$(pwd)             # Local development imports
 ## Performance Requirements
 
 - **Core Package**: <2MB (from ~8MB in v4.0.x)
-- **Regex Engine**: 150x+ faster than spaCy (currently 190x)
+- **Regex Engine**: 100x+ faster than spaCy (103–170x measured across payloads; see `benchmarks/`)
 - **GLiNER Engine**: 25x+ faster than spaCy (currently 32x)
 - **Memory Usage**: Graceful handling of large texts (1MB+ chunks)
 - **Model Loading**: Cache GLiNER models to avoid repeated downloads

CHANGELOG.MD

@@ -1,5 +1,43 @@
 # ChangeLog
 
+## [Unreleased]
+
+## [2026-07-05]
+
+### `datafog-python` [4.8.0]
+
+#### Fixed
+
+- **Redaction token numbering now follows document order**: `redact()`
+  previously assigned per-type counters while replacing spans right-to-left,
+  so with multiple entities of the same type the *last* occurrence received
+  `_1` (two emails redacted as `[EMAIL_2] ... [EMAIL_1]`). Tokens for the
+  `token` and `pseudonymize` strategies are now numbered left-to-right, so
+  the first occurrence is always `_1`. This changes redacted output text for
+  multi-entity inputs; `RedactResult.entities` is now also ordered by
+  document position (ascending) instead of descending. Replacement values
+  in `mapping` are now always read from the original input text, fixing a
+  case where an already-inserted token could leak into a mapping value when
+  entity spans overlapped.
+- **`redact()` now resolves overlapping and duplicate entity spans**:
+  `redact()` is public API and accepts arbitrary entity lists, but applied
+  every span verbatim — overlapping spans (e.g. a phone number and a
+  sub-span of it) corrupted the output with stray fragments like
+  `[PHONE_1]]`. For each group of overlapping spans only one is now
+  redacted: the longest, breaking ties by entity type priority and then
+  confidence — the same suppression rule the regex scan pipeline already
+  used. Suppressed spans are omitted from `RedactResult.entities` and
+  `mapping`. Output for non-overlapping entity lists (including everything
+  produced by `scan()`) is unchanged.
+- **`mask` strategy mapping no longer collides on same-length values**:
+  the mask replacement (`***…`) previously keyed `mapping` directly, so two
+  distinct values of the same length produced the same key and only one
+  original survived. Mask mappings are now keyed by per-type indexed keys
+  in document order (`[EMAIL_MASK_1]`, `[EMAIL_MASK_2]`), preserving every
+  original value. Consumers that looked up mask mappings by the mask string
+  must switch to the indexed keys; `token`, `hash`, and `pseudonymize`
+  mapping keys are unchanged.
+
 ## [2026-07-02]
 
 ### `datafog-python` [4.7.0]

README.md

@@ -19,8 +19,8 @@ values never echoed into logs or transcripts:
 
 - **Claude Code hook** (`datafog-hook`): gates agent tool calls (shell
   commands, web requests, file writes, MCP tools) and warns the model when
-  prompts or tool results carry PII. ~70ms per invocation including process
-  startup. Easiest install is the
+  prompts or tool results carry PII. ~70–90ms per invocation including
+  process startup. Easiest install is the
   [Claude Code plugin](https://github.com/DataFog/datafog-claude-plugin):
 
   ```
@@ -32,7 +32,8 @@ values never echoed into logs or transcripts:
 
 - **LiteLLM guardrail** (`DataFogGuardrail`): redacts or blocks PII in
   requests and responses at the gateway, for any LiteLLM-proxied provider.
-  In-process (~31µs per request), no sidecar service. Setup:
+  In-process (~40µs per message scanned; a request clears the guardrail in
+  well under a millisecond), no sidecar service. Setup:
   [examples/litellm_guardrail/](examples/litellm_guardrail/).
 
 Both default to the high-precision entity set (`EMAIL`, `PHONE`,
@@ -43,6 +44,10 @@ unix timestamps matching as phone numbers) — available in both adapters and
 the API. Presidio-style entity names (`EMAIL_ADDRESS`, `PHONE_NUMBER`,
 `US_SSN`) are accepted as aliases for easy migration.
 
+Every performance number above is reproducible with one command —
+methodology, pinned payloads, and comparisons against Presidio and spaCy
+NER live in [benchmarks/](benchmarks/).
+
 ## Installation
 
 ```bash

ROADMAP.md

@@ -0,0 +1,8 @@
+# DataFog Roadmap
+
+The roadmap — current release status (4.7.x) and v5.0.0 plans — lives in
+[docs/roadmap.rst](docs/roadmap.rst).
+
+Roadmap priorities are shaped by user feedback — open a
+[GitHub issue](https://github.com/DataFog/datafog-python/issues) or join
+the [Discord](https://discord.gg/bzDth394R4).

benchmarks/README.md

@@ -0,0 +1,151 @@
+# DataFog benchmarks
+
+Every performance number DataFog publishes should be reproducible by a
+skeptic with one command. This directory is that command:
+
+```bash
+pip install -e . -r benchmarks/requirements.txt
+python -m spacy download en_core_web_sm   # comparison suites only
+python benchmarks/run.py
+```
+
+Suites that are missing optional dependencies are skipped with an install
+hint — `python benchmarks/run.py --suite core,hook` runs with nothing but
+datafog itself installed. `--json results.json` writes machine-readable
+output; `--quick` is a fast smoke run (fewer repeats, large payloads
+skipped).
+
+## Methodology
+
+- **In-process timings** use stdlib `timeit`: `Timer.autorange()` picks an
+  inner loop count so each timed repeat runs ≥0.2s, then 5 repeats (3 in
+  comparison suites) are reported as median ± stdev. No mocking — every
+  call goes through the same public code path a user hits.
+- **The hook suite measures wall-clock subprocess time**: a fresh
+  `datafog-hook` process per iteration, JSON payload on stdin, exactly as
+  Claude Code invokes it. 3 warmup runs, 30 timed runs, median and p90
+  reported. Python interpreter startup is _included_ — that is the honest
+  cost of a per-tool-call hook.
+- **Pinned payloads** live in [payloads/](payloads/) and are checked in,
+  not generated at runtime. [payloads/manifest.json](payloads/manifest.json)
+  records the entity counts the regex engine must find in each payload; the
+  runner verifies those counts _before_ timing anything, so the suite fails
+  loudly rather than silently benchmarking an engine that stopped
+  detecting. All PII values are industry-standard synthetic test data
+  (reserved `.invalid`/`example.com` domains, fictional 555-01XX phone
+  numbers, public Luhn-valid test cards, SSA example SSNs).
+
+| Payload                   | Size   | Content                                                                    |
+| ------------------------- | ------ | -------------------------------------------------------------------------- |
+| `core_1kb_dense.txt`      | 1.2 KB | support ticket, 8 entities (PII-dense)                                     |
+| `core_10kb_mixed.txt`     | 10 KB  | business document, 12 entities in prose                                    |
+| `core_100kb_sparse.txt`   | 100 KB | machine logs, 18 entities among UUIDs/timestamps (false-positive pressure) |
+| `chat_request_clean.json` | —      | 2-message LiteLLM chat body, no PII                                        |
+| `chat_request_pii.json`   | —      | 2-message LiteLLM chat body, email + card + phone                          |
+| `hook_pretooluse.json`    | —      | Claude Code `PreToolUse` payload, `curl` command carrying PII              |
+
+All suites use the production default entity set
+(`EMAIL, PHONE, CREDIT_CARD, SSN`).
+
+## Reference results
+
+Apple M5 Pro, macOS, CPython 3.13, datafog 4.7.0, litellm 1.91.0,
+presidio-analyzer 2.2.363, spaCy 3.8 (`en_core_web_sm`). Medians of the
+full (non-quick) run; expect different absolute numbers on different
+hardware, but the ratios and orders of magnitude should hold.
+
+### core — `datafog.scan` / `datafog.redact`, regex engine
+
+| Operation            | Median  | Throughput |
+| -------------------- | ------- | ---------- |
+| scan 1.2 KB dense    | 231 µs  | 5.3 MB/s   |
+| redact 1.2 KB dense  | 240 µs  | 5.1 MB/s   |
+| scan 10 KB mixed     | 1.20 ms | 8.6 MB/s   |
+| redact 10 KB mixed   | 1.19 ms | 8.6 MB/s   |
+| scan 100 KB sparse   | 11.9 ms | 8.4 MB/s   |
+| redact 100 KB sparse | 12.1 ms | 8.3 MB/s   |
+
+### guardrail — `DataFogGuardrail` (LiteLLM), in-process
+
+| Operation                                              | Median |
+| ------------------------------------------------------ | ------ |
+| single message redact                                  | 43 µs  |
+| `pre_call`, clean 2-message request                    | 118 µs |
+| `pre_call`, PII 2-message request (redact)             | 222 µs |
+| event-loop dispatch (harness overhead, included above) | 27 µs  |
+
+The PII-request figure includes litellm's own guardrail-intervention
+logging (~85 µs), not just detection. In a live proxy the event loop is
+already running, so the ~27 µs dispatch overhead in the two per-request
+rows is an artifact of benchmarking with `run_until_complete`.
+
+### hook — `datafog-hook` (Claude Code)
+
+| Operation                                   | Median                               |
+| ------------------------------------------- | ------------------------------------ |
+| end-to-end subprocess, incl. Python startup | 69–89 ms across runs (p90 ~77–96 ms) |
+| the scan itself (in-process)                | 75 µs                                |
+
+The end-to-end cost is ~99.9% Python interpreter + import startup; the
+scan is microseconds. Cold (first-ever) invocations can spike to several
+hundred ms while the OS warms caches — the warmup runs exclude that, the
+p90 shows steady-state spread.
+
+### Comparisons — same payloads, same four entity types
+
+Both comparison targets are pinned to `en_core_web_sm`, the _smallest_
+English spaCy model — deliberately favorable to them (smaller model =
+faster inference), so these ratios are lower bounds. Model/engine setup
+time (~0.1–1 s) is excluded from per-scan figures. Presidio runs in-process
+via `AnalyzerEngine` with a documented `NlpEngineProvider` config — no
+sidecar, no network hop, which again flatters Presidio relative to its
+usual proxy deployment.
+
+| Payload       | datafog regex | Presidio | spaCy NER | datafog vs Presidio | datafog vs spaCy |
+| ------------- | ------------- | -------- | --------- | ------------------- | ---------------- |
+| 1.2 KB dense  | 222–251 µs    | 22.8 ms  | 28.6 ms   | 103x                | 114x             |
+| 10 KB mixed   | 1.2 ms        | 174 ms   | 171 ms    | 148x                | 140x             |
+| 100 KB sparse | 11.9–12.2 ms  | 2.02 s   | 1.60 s    | 170x                | 131x             |
+
+Speed is not the only axis: the runner records what each engine actually
+detected on each payload (see the `detected` metadata in the output).
+Recall differs in both directions — e.g. Presidio's email recognizer does
+not match addresses on the reserved `.invalid` domain, and context-based
+scoring drops some entities the regex engine finds; conversely NER models
+detect names and locations the regex engine does not target at all. The
+comparison is a fair _speed_ comparison on identical inputs and entity
+types, not an accuracy study.
+
+## How these map to published claims
+
+- **"~31µs per request" (LiteLLM guardrail, project README /
+  datafog.ai)** — not reproduced _as worded_. ~31–43 µs is the cost of
+  scanning/redacting **one short message**; a full 2-message request
+  through `async_pre_call_hook` is ~120 µs clean / ~220 µs with redaction
+  and litellm logging. Still 100–1000x below a sidecar's network hop, but
+  the claim should be reworded to per-message, e.g. _"~40µs per message —
+  a request clears the guardrail in well under a millisecond"_. Reproduce:
+  `python benchmarks/run.py --suite guardrail`.
+- **"~70ms per invocation including process startup" (Claude Code
+  hook)** — reproduced: 69–89 ms median across runs on this machine
+  (~70 ms idle, higher under load), dominated by interpreter startup.
+  _"~70–90ms"_ (or "under 100ms") is the defensible phrasing. Reproduce: `python benchmarks/run.py --suite hook`.
+- **"190x performance advantage" (PyPI tagline)** — not reproduced at
+  190x by this suite. Measured: 103–170x vs Presidio and 114–140x vs spaCy
+  NER, varying by payload. The historical 190x came from a different
+  (13.3 KB, entity-dense) document and engine configuration. Honest
+  phrasings this suite supports: _"100x+ faster than NER-based PII
+  detection"_ or _"up to 170x faster than Presidio on identical
+  payloads"_. Reproduce: `python benchmarks/run.py --suite spacy,presidio`.
+
+## Fairness notes / known limitations
+
+- Single-threaded, single-machine, synthetic payloads. Real chat traffic
+  and real documents will differ; the payloads are designed to bracket the
+  realistic range (dense, mixed, sparse).
+- `timeit` medians hide GC pauses and tail latency by design; the hook
+  suite's p90 is the only tail-latency figure here.
+- The comparison suites measure the engines _as configured here_
+  (documented model, in-process, setup excluded). Different Presidio
+  recognizer registries or larger spaCy models change both speed and
+  recall — in Presidio's favor on recall, against it on speed.