Skip to content

docs: add public v5.0.0 roadmap#151

Merged
sidmohan0 merged 1 commit into
devfrom
docs/v5-roadmap
Jul 2, 2026
Merged

docs: add public v5.0.0 roadmap#151
sidmohan0 merged 1 commit into
devfrom
docs/v5-roadmap

Conversation

@sidmohan0

Copy link
Copy Markdown
Contributor

What this is

A public-facing ROADMAP.md at the repo root laying out the v5.0.0 direction, published now that v4.5.0 is live on PyPI. Internal planning lives in the Linear project (DataFog Python v5.0.0: Offline PII Firewall for AI Apps, epics DFPY-110–115, target July 8); this document is the community-visible summary.

Also removes a stale Roadmap.md entry from .gitignore (no such private file exists) so the roadmap can live at the root where people expect it.

The v5 thesis in one line

DataFog is independently benchmarked as the fastest Python PII library by orders of magnitude; v5 invests that speed advantage into precision, EU coverage, and a leaner package.

The five themes

  1. Precision — checksum validators (Luhn, IBAN mod-97, SSN rules, IP plausibility), confidence scores, and strictness presets. Directly answers the main criticism in third-party evaluations (fast but noisy — false positives like number sequences flagged as IPs).
  2. EU languages — generalize the German locale support shipped in 4.5.0 into a pattern-pack system; EU-wide entities (IBAN, VAT IDs, national IDs) first, then FR/ES/IT/NL/PL across v5.x.
  3. Leaner package — drop the Donut/transformers OCR path, spin out the PySpark wrapper, delete legacy duplicate modules; CI-enforced wheel-size and import-time budgets.
  4. Pipeline embeddability — batch/iterator APIs plus recipes for Kafka, Vector/Fluent Bit, and OpenTelemetry, instead of shipping transport connectors.
  5. Vault-friendly anonymization — deterministic tokenization with exportable mappings, slotting into vault+token architectures.

Open questions for discussion

  • Cut line: proposed v5.0.0 core is themes 1 + 3 + API promotion (scan/redact/protect primary), with themes 2/4/5 landing across v5.x. Is that the right minimal breaking release?
  • Timeline: July 8 target — realistic for the core cut, or should precision benchmarks gate the stable and push it out?
  • Spin-outs: do OCR and Spark deserve companion packages (datafog-ocr, datafog-spark) or just documented recipes?
  • Positioning: the precision/recall benchmark vs Presidio is a marketing asset as much as an engineering gate — publish alongside the release or before it?

Test plan

  • Docs-only change; verify ROADMAP.md renders correctly on GitHub
  • Confirm no untracked private Roadmap.md files exist in contributor clones before the gitignore removal lands

Removes the stale Roadmap.md gitignore entry (no such private file
exists) so the public roadmap can live at the repo root.
@sidmohan0 sidmohan0 merged commit a1feddb into dev Jul 2, 2026
26 checks passed
sidmohan0 added a commit that referenced this pull request Jul 2, 2026
This reverts commit a1feddb, reversing
changes made to 2fbce0a.
@sidmohan0 sidmohan0 mentioned this pull request Jul 2, 2026
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant