diff --git a/AGENTS.md b/AGENTS.md index 4b43fe1f..1a174106 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -13,19 +13,19 @@ ## Current Project Status -**Stable version: 4.4.0** +**Stable version: 4.8.0** -**Development version: 4.4.0a5** +**Development version: 4.8.0** -**Next minor target: 4.5.0** +**Next major target: 5.0.0** ### ✅ Recently Completed (Latest) +- **Redaction Correctness**: Document-order token numbering, overlap-safe span handling, and stable mask mappings +- **Allowlists and Typing**: Exact and regex allowlists, Presidio-style aliases, and `py.typed` +- **Agent & Gateway Firewall**: Claude Code hook and LiteLLM guardrail adapters - **GLiNER Integration**: Modern NER engine with PII-specialized models -- **Smart Cascading**: Intelligent regex → GLiNER → spaCy progression -- **Enhanced CLI**: Model management with `--engine` flags - **Performance Validation**: 100x+ regex speedup reproducible via `benchmarks/`; 32x GLiNER -- **CI/CD Consolidation**: 7 workflows → 3 (ci, release, benchmark) ## Quick Development Setup @@ -41,7 +41,7 @@ pre-commit install # 3. Install ML extras for advanced features pip install -e ".[nlp]" # spaCy -pip install -e ".[nlp-advanced]" # GLiNER (NEW) +pip install -e ".[nlp-advanced]" # GLiNER pip install -e ".[all]" # Everything # 4. Verify installation @@ -89,7 +89,7 @@ pip install datafog[ocr] # Image processing pip install datafog[all] # Everything ``` -## GLiNER Integration (NEW) +## GLiNER Integration ### Overview diff --git a/README.md b/README.md index 4f3588f0..b974e48a 100644 --- a/README.md +++ b/README.md @@ -6,14 +6,14 @@ It provides: - Fast structured PII detection via regex - An offline PII firewall for AI agents: a Claude Code hook and a LiteLLM - gateway guardrail (new in 4.6) + gateway guardrail (introduced in 4.6) - Optional NER support via spaCy and GLiNER - A simple agent-oriented API for LLM applications - Backward-compatible `DataFog` and `TextService` classes -## Agent & Gateway Firewall (4.6) +## Agent & Gateway Firewall -DataFog 4.6 adds two ready-made enforcement points that catch PII at the +DataFog includes two ready-made enforcement points that catch PII at the moment it would leave your machine — offline, in microseconds, with matched values never echoed into logs or transcripts: @@ -72,8 +72,8 @@ pip install datafog[all] Python 3.13 support is certified for the core SDK, CLI, `nlp`, `nlp-advanced`, and `ocr` install profiles. Donut OCR still requires a model -that is available locally before runtime use. `distributed` and `all` are not -newly certified on Python 3.13 in the 4.x line. +that is available locally before runtime use. `distributed` and `all` remain +optional, heavier profiles and are not part of the lightweight core path. ## Quick Start diff --git a/ROADMAP.md b/ROADMAP.md index 0850d901..af9b26a0 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -1,6 +1,6 @@ # DataFog Roadmap -The roadmap — current release status (4.7.x) and v5.0.0 plans — lives in +The roadmap — current release status (4.8.x) and v5.0.0 plans — lives in [docs/roadmap.rst](docs/roadmap.rst). Roadmap priorities are shaped by user feedback — open a diff --git a/benchmarks/README.md b/benchmarks/README.md index 93ec4923..aaea9470 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -49,7 +49,7 @@ All suites use the production default entity set ## Reference results -Apple M5 Pro, macOS, CPython 3.13, datafog 4.7.0, litellm 1.91.0, +Apple M5 Pro, macOS, CPython 3.13, datafog 4.8.0, litellm 1.91.0, presidio-analyzer 2.2.363, spaCy 3.8 (`en_core_web_sm`). Medians of the full (non-quick) run; expect different absolute numbers on different hardware, but the ratios and orders of magnitude should hold. @@ -116,28 +116,6 @@ detect names and locations the regex engine does not target at all. The comparison is a fair _speed_ comparison on identical inputs and entity types, not an accuracy study. -## How these map to published claims - -- **"~31µs per request" (LiteLLM guardrail, project README / - datafog.ai)** — not reproduced _as worded_. ~31–43 µs is the cost of - scanning/redacting **one short message**; a full 2-message request - through `async_pre_call_hook` is ~120 µs clean / ~220 µs with redaction - and litellm logging. Still 100–1000x below a sidecar's network hop, but - the claim should be reworded to per-message, e.g. _"~40µs per message — - a request clears the guardrail in well under a millisecond"_. Reproduce: - `python benchmarks/run.py --suite guardrail`. -- **"~70ms per invocation including process startup" (Claude Code - hook)** — reproduced: 69–89 ms median across runs on this machine - (~70 ms idle, higher under load), dominated by interpreter startup. - _"~70–90ms"_ (or "under 100ms") is the defensible phrasing. Reproduce: `python benchmarks/run.py --suite hook`. -- **"190x performance advantage" (PyPI tagline)** — not reproduced at - 190x by this suite. Measured: 103–170x vs Presidio and 114–140x vs spaCy - NER, varying by payload. The historical 190x came from a different - (13.3 KB, entity-dense) document and engine configuration. Honest - phrasings this suite supports: _"100x+ faster than NER-based PII - detection"_ or _"up to 170x faster than Presidio on identical - payloads"_. Reproduce: `python benchmarks/run.py --suite spacy,presidio`. - ## Fairness notes / known limitations - Single-threaded, single-machine, synthetic payloads. Real chat traffic diff --git a/docs/contributing.rst b/docs/contributing.rst index b1b58777..7f00e029 100644 --- a/docs/contributing.rst +++ b/docs/contributing.rst @@ -1,23 +1,23 @@ -======================================= -Contributor Setup And 4.5 Release Flow -======================================= +================================== +Contributor Setup And Release Flow +================================== -This page is the contributor runbook for DataFog 4.5 work. It is meant for +This page is the contributor runbook for DataFog 4.x work. It is meant for humans and agents preparing local changes, choosing verification commands, and -understanding where the 4.5 release boundary sits. +understanding where the current release boundary sits. Version Frame ============= Current release planning uses this frame: -* Stable package release: ``4.4.0``. -* Current development package version: ``4.4.0a5``. -* Next minor target: ``4.5.0``. +* Stable package release: ``4.8.0``. +* Current development package version: ``4.8.0``. +* Next major target: ``5.0.0``. -Do not bump routine feature, documentation, or cleanup branches directly to -``4.5.0``. Keep the version stable during local release-prep work, then handle -the final version and release-note alignment in the release-readiness slice. +Do not bump routine feature, documentation, or cleanup branches directly to a +future release number. Keep the version stable during local work, then handle +final version and release-note alignment in the dedicated release slice. Python Environments =================== @@ -25,8 +25,8 @@ Python Environments DataFog currently declares support for Python ``>=3.10,<3.14``. The CI matrix tests core, NLP, and NLP-advanced installs on Python 3.10, 3.11, 3.12, and 3.13. OCR profile smoke checks also run on Python 3.13 with system Tesseract -installed. Distributed and all-profile Python 3.13 validation remain outside -the 4.5 support claim. +installed. Distributed and all-profile installs remain heavier optional +profiles rather than the lightweight core support claim. Create one virtual environment per Python version when you need to compare profiles locally: @@ -171,41 +171,35 @@ Live Modules ============ Use :doc:`live-module-map` before changing core package structure. It lists the -live 4.5 modules for each concept and the historical ``*_lean`` and +live 4.x modules for each concept and the historical ``*_lean`` and ``*_original`` files that are kept only as non-live compatibility/audit artifacts. -4.5 Release Flow -================ +Release Flow +============ -The 4.5 work lands as focused pull requests into ``dev``. Keep feature and docs -branches narrow, and avoid mixing local cleanup, external PR review, and final -release mechanics in one branch. +4.x and v5 preparation work lands as focused pull requests into ``dev``. Keep +feature and docs branches narrow, and avoid mixing local cleanup, external PR +review, and final release mechanics in one branch. -The release flow for 4.5 is: +The release flow is: -1. Land the local release-prep baseline and follow-up cleanup/docs slices. -2. Review the external German regex PR after the local release-prep baseline is - in place. -3. Integrate German regex support only if review says it fits the 4.5 - lightweight text screening thesis. -4. Validate optional Python 3.13 profiles before claiming support beyond core - SDK and CLI. -5. Prepare release readiness with :doc:`v45-release-readiness`: changelog and - release notes, package checks, docs build, CI state, and version alignment. -6. Bump or override the final stable release to ``4.5.0`` only during the - release-readiness and stable-release path. +1. Land focused feature, fix, or docs slices into ``dev``. +2. Update the changelog, release notes, package checks, docs build, CI state, + and version alignment in a dedicated release-readiness change. +3. Validate optional Python 3.13 profiles before expanding support claims. +4. Bump or override the final stable release version only during the + stable-release path. The current release workflow strips prerelease suffixes from the package -version unless a manual stable ``version_override`` is provided. For the final -4.5 stable release, use a dedicated release-readiness change or the stable -workflow override so the published version is ``4.5.0`` rather than another -``4.4.0`` prerelease line. +version unless a manual stable ``version_override`` is provided. Use a +dedicated release-readiness change or the stable workflow override when the +published version should move to a new stable line. External PR Boundary ==================== -The external German PII regex PR belongs after local baseline cleanup. Review -it as a 4.5 candidate, not as a v5 planning shortcut. If accepted, adapt it in -the German regex integration slice with tests, documentation of locale -coverage, and no new dependency burden on the core path. +Review external PII regex PRs against the current lightweight text screening +thesis, not as v5 planning shortcuts. If accepted, adapt them with tests, +documentation of locale coverage, and no new dependency burden on the core +path. diff --git a/docs/getting-started.rst b/docs/getting-started.rst index eeeeb46d..cf6efca7 100644 --- a/docs/getting-started.rst +++ b/docs/getting-started.rst @@ -123,11 +123,11 @@ The CLI core path is text-first: Image commands are optional. Install ``datafog[ocr]`` for local OCR and ``datafog[web,ocr]`` when the CLI needs to download image inputs. -What 4.5 Is Not +What 4.x Is Not =============== -DataFog 4.5 prepares the package for future middleware use cases, but it does -not ship dedicated Sentry, OpenTelemetry, logging-framework, or cloud DLP +DataFog 4.x prepares the package for broader middleware use cases, but it does +not yet ship dedicated Sentry, OpenTelemetry, logging-framework, or cloud DLP adapters. Those integrations are future-facing work built on the same core text screening path. @@ -137,4 +137,4 @@ Next Pages * :doc:`python-sdk` documents the Python API surface. * :doc:`cli` documents command-line usage. * :doc:`optional-surfaces` documents OCR and Spark install notes. -* :doc:`roadmap` explains how 4.5 leads toward later middleware work. +* :doc:`roadmap` explains how the 4.x line leads toward later middleware work. diff --git a/docs/roadmap.rst b/docs/roadmap.rst index d350801b..080a6b0f 100644 --- a/docs/roadmap.rst +++ b/docs/roadmap.rst @@ -2,7 +2,7 @@ Release Roadmap ================ -Where DataFog is today (4.7.x) and where it is going (v5.0.0). The 4.x +Where DataFog is today (4.8.x) and where it is going (v5.0.0). The 4.x line delivered the lightweight-core architecture and, from 4.6.0 on, an offline PII firewall for AI agents and gateways. The v5 cycle turns that foundation into the fastest, easiest offline PII firewall for AI apps, @@ -12,13 +12,13 @@ logs, and datasets. :local: :depth: 1 -Current status — 4.7.x +Current status — 4.8.x ---------------------- -DataFog ``4.7.0`` is the current stable release (July 2026). Patch -releases on the 4.7.x line focus on detection precision — reducing -SSN/phone false positives surfaced by real-world agent traffic — while -v5.0.0 is prepared. +DataFog ``4.8.0`` is the current stable release (July 2026). Patch +releases on the 4.8.x line focus on redaction correctness, detection +precision, and keeping the 4.x agent/gateway firewall stable while v5.0.0 +is prepared. Every published performance number is reproducible with one command: ``python benchmarks/run.py`` (see ``benchmarks/`` for methodology, @@ -112,6 +112,17 @@ Both default to the high-precision entity set (``EMAIL``, ``PHONE``, 4.7.x patch releases: SSN and phone precision fixes (hex-string and no-dash false positives). +✅ 4.8.0 — Redaction Correctness (Released July 2026) +------------------------------------------------------ + +* **Document-order redaction tokens**: repeated entities are numbered from + left to right, so the first email in a document becomes ``[EMAIL_1]``. +* **Overlap-safe redaction**: public ``redact()`` calls now suppress + overlapping and duplicate spans before applying replacements, avoiding + corrupted output fragments. +* **Mask mapping preservation**: same-length values no longer collide in + ``mask`` strategy mappings; each masked value receives a stable indexed key. + v5.0.0 — Offline PII Firewall for AI Apps (In Progress) -------------------------------------------------------- diff --git a/examples/quick_start.ipynb b/examples/quick_start.ipynb index ea7aa03c..d6381847 100644 --- a/examples/quick_start.ipynb +++ b/examples/quick_start.ipynb @@ -16,7 +16,7 @@ { "cell_type": "markdown", "metadata": {}, - "source": "# DataFog Quick Start Guide\\n\\n> **📦 Version Requirement**: This guide is for DataFog v4.2.0 and above\\n> \\n> ✅ **New in v4.2.0**: GLiNER integration, smart cascading, and enhanced performance\\n\\nWelcome to DataFog! This notebook demonstrates how to get started with DataFog's fast PII detection and anonymization capabilities.\\n\\n## What makes DataFog special?\\n\\n- **🚀 Ultra-Fast**: 190x faster than spaCy for structured PII, 32x faster with GLiNER\\n- **🪶 Lightweight**: <2MB core package with optional ML extras\\n- **🧠 Smart Engines**: Choose from regex, GLiNER, spaCy, or smart cascading\\n- **📦 Production Ready**: Comprehensive testing and performance validation\"", + "source": "# DataFog Quick Start Guide\\n\\n> **📦 Version Requirement**: This guide is current for DataFog v4.8.0 and above\\n> \\n> ✅ **4.x Highlights**: GLiNER integration, smart cascading, agent guardrails, allowlists, and redaction correctness fixes\\n\\nWelcome to DataFog! This notebook demonstrates how to get started with DataFog's fast PII detection and anonymization capabilities.\\n\\n## What makes DataFog special?\\n\\n- **🚀 Ultra-Fast**: 190x faster than spaCy for structured PII, 32x faster with GLiNER\\n- **🪶 Lightweight**: <2MB core package with optional ML extras\\n- **🧠 Smart Engines**: Choose from regex, GLiNER, spaCy, or smart cascading\\n- **📦 Production Ready**: Comprehensive testing and performance validation\"", "outputs": [] }, { @@ -624,4 +624,4 @@ }, "nbformat": 4, "nbformat_minor": 4 -} \ No newline at end of file +} diff --git a/setup_lean.py b/setup_lean.py index 3d8296bb..c3b8e15e 100644 --- a/setup_lean.py +++ b/setup_lean.py @@ -1,6 +1,6 @@ from setuptools import find_packages, setup -# Historical shadow packaging snapshot. The live DataFog 4.5 packaging input is +# Historical shadow packaging snapshot. The live DataFog packaging input is # setup.py. Do not use this file for builds, releases, or dependency changes. # Read README for the long description diff --git a/setup_original.py b/setup_original.py index d70db319..23692041 100644 --- a/setup_original.py +++ b/setup_original.py @@ -1,6 +1,6 @@ from setuptools import find_packages, setup -# Historical shadow packaging snapshot. The live DataFog 4.5 packaging input is +# Historical shadow packaging snapshot. The live DataFog packaging input is # setup.py. Do not use this file for builds, releases, or dependency changes. # Read README for the long description