diff --git a/.codecov.yml b/.codecov.yml index a052f98d..8b3c36b1 100644 --- a/.codecov.yml +++ b/.codecov.yml @@ -1 +1,10 @@ comment: no + +codecov: + notify: + # The CI test matrix uploads 12 coverage reports (4 Python versions x 3 + # install profiles). Only the nlp-advanced profile installs litellm, so + # the guardrail adapter's lines look uncovered until those uploads land. + # Waiting for all builds prevents the recurring interim-red patch status. + after_n_builds: 12 + wait_for_ci: true diff --git a/.gitignore b/.gitignore index 1316a1f3..cf11a42c 100644 --- a/.gitignore +++ b/.gitignore @@ -69,5 +69,4 @@ docs/* */**/__pycache__/ notes/benchmarking_notes.md -Roadmap.md notes/* diff --git a/AGENTS.md b/AGENTS.md index a46c4402..4b43fe1f 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,7 +6,7 @@ ## Core Value Proposition -- **Ultra-Fast Performance**: 190x faster than spaCy for structured PII, 32x faster with GLiNER +- **Ultra-Fast Performance**: 100x+ faster than spaCy/Presidio for structured PII (reproduce: `python benchmarks/run.py`), 32x faster with GLiNER - **Lightweight Core**: <2MB package with optional ML extras - **Modern Engine Options**: Regex, GLiNER, spaCy, and smart cascading - **Production Ready**: Comprehensive testing, CI/CD, and performance validation @@ -24,7 +24,7 @@ - **GLiNER Integration**: Modern NER engine with PII-specialized models - **Smart Cascading**: Intelligent regex → GLiNER → spaCy progression - **Enhanced CLI**: Model management with `--engine` flags -- **Performance Validation**: 190x regex, 32x GLiNER benchmarks confirmed +- **Performance Validation**: 100x+ regex speedup reproducible via `benchmarks/`; 32x GLiNER - **CI/CD Consolidation**: 7 workflows → 3 (ci, release, benchmark) ## Quick Development Setup @@ -56,7 +56,7 @@ python -c "from datafog.services.text_service import TextService; print('✅ All from datafog.services.text_service import TextService # Core engines (always available) -regex_service = TextService(engine="regex") # 190x faster, structured PII +regex_service = TextService(engine="regex") # 100x+ faster, structured PII # ML engines (require extras) gliner_service = TextService(engine="gliner") # 32x faster, modern NER @@ -69,12 +69,12 @@ auto_service = TextService(engine="auto") # Legacy: regex→spaCy ### Performance Comparison (Validated) -| Engine | Speed vs spaCy | Accuracy | Use Case | Install | -| -------- | --------------- | ----------------- | --------------------------- | ---------------- | -| `regex` | **190x faster** | High (structured) | Emails, phones, SSNs | Core only | -| `gliner` | **32x faster** | Very High | Modern NER, custom entities | `[nlp-advanced]` | -| `spacy` | 1x (baseline) | Good | Traditional NLP | `[nlp]` | -| `smart` | **60x faster** | Highest | Best balance | `[nlp-advanced]` | +| Engine | Speed vs spaCy | Accuracy | Use Case | Install | +| -------- | ---------------- | ----------------- | --------------------------- | ---------------- | +| `regex` | **100x+ faster** | High (structured) | Emails, phones, SSNs | Core only | +| `gliner` | **32x faster** | Very High | Modern NER, custom entities | `[nlp-advanced]` | +| `spacy` | 1x (baseline) | Good | Traditional NLP | `[nlp]` | +| `smart` | **60x faster** | Highest | Best balance | `[nlp-advanced]` | ### Dependency Strategy @@ -281,7 +281,7 @@ export PYTHONPATH=$(pwd) # Local development imports ## Performance Requirements - **Core Package**: <2MB (from ~8MB in v4.0.x) -- **Regex Engine**: 150x+ faster than spaCy (currently 190x) +- **Regex Engine**: 100x+ faster than spaCy (103–170x measured across payloads; see `benchmarks/`) - **GLiNER Engine**: 25x+ faster than spaCy (currently 32x) - **Memory Usage**: Graceful handling of large texts (1MB+ chunks) - **Model Loading**: Cache GLiNER models to avoid repeated downloads diff --git a/CHANGELOG.MD b/CHANGELOG.MD index b51971c0..a2655567 100644 --- a/CHANGELOG.MD +++ b/CHANGELOG.MD @@ -1,5 +1,43 @@ # ChangeLog +## [Unreleased] + +## [2026-07-05] + +### `datafog-python` [4.8.0] + +#### Fixed + +- **Redaction token numbering now follows document order**: `redact()` + previously assigned per-type counters while replacing spans right-to-left, + so with multiple entities of the same type the *last* occurrence received + `_1` (two emails redacted as `[EMAIL_2] ... [EMAIL_1]`). Tokens for the + `token` and `pseudonymize` strategies are now numbered left-to-right, so + the first occurrence is always `_1`. This changes redacted output text for + multi-entity inputs; `RedactResult.entities` is now also ordered by + document position (ascending) instead of descending. Replacement values + in `mapping` are now always read from the original input text, fixing a + case where an already-inserted token could leak into a mapping value when + entity spans overlapped. +- **`redact()` now resolves overlapping and duplicate entity spans**: + `redact()` is public API and accepts arbitrary entity lists, but applied + every span verbatim — overlapping spans (e.g. a phone number and a + sub-span of it) corrupted the output with stray fragments like + `[PHONE_1]]`. For each group of overlapping spans only one is now + redacted: the longest, breaking ties by entity type priority and then + confidence — the same suppression rule the regex scan pipeline already + used. Suppressed spans are omitted from `RedactResult.entities` and + `mapping`. Output for non-overlapping entity lists (including everything + produced by `scan()`) is unchanged. +- **`mask` strategy mapping no longer collides on same-length values**: + the mask replacement (`***…`) previously keyed `mapping` directly, so two + distinct values of the same length produced the same key and only one + original survived. Mask mappings are now keyed by per-type indexed keys + in document order (`[EMAIL_MASK_1]`, `[EMAIL_MASK_2]`), preserving every + original value. Consumers that looked up mask mappings by the mask string + must switch to the indexed keys; `token`, `hash`, and `pseudonymize` + mapping keys are unchanged. + ## [2026-07-02] ### `datafog-python` [4.7.0] diff --git a/README.md b/README.md index 965a5808..4f3588f0 100644 --- a/README.md +++ b/README.md @@ -19,8 +19,8 @@ values never echoed into logs or transcripts: - **Claude Code hook** (`datafog-hook`): gates agent tool calls (shell commands, web requests, file writes, MCP tools) and warns the model when - prompts or tool results carry PII. ~70ms per invocation including process - startup. Easiest install is the + prompts or tool results carry PII. ~70–90ms per invocation including + process startup. Easiest install is the [Claude Code plugin](https://github.com/DataFog/datafog-claude-plugin): ``` @@ -32,7 +32,8 @@ values never echoed into logs or transcripts: - **LiteLLM guardrail** (`DataFogGuardrail`): redacts or blocks PII in requests and responses at the gateway, for any LiteLLM-proxied provider. - In-process (~31µs per request), no sidecar service. Setup: + In-process (~40µs per message scanned; a request clears the guardrail in + well under a millisecond), no sidecar service. Setup: [examples/litellm_guardrail/](examples/litellm_guardrail/). Both default to the high-precision entity set (`EMAIL`, `PHONE`, @@ -43,6 +44,10 @@ unix timestamps matching as phone numbers) — available in both adapters and the API. Presidio-style entity names (`EMAIL_ADDRESS`, `PHONE_NUMBER`, `US_SSN`) are accepted as aliases for easy migration. +Every performance number above is reproducible with one command — +methodology, pinned payloads, and comparisons against Presidio and spaCy +NER live in [benchmarks/](benchmarks/). + ## Installation ```bash diff --git a/ROADMAP.md b/ROADMAP.md new file mode 100644 index 00000000..0850d901 --- /dev/null +++ b/ROADMAP.md @@ -0,0 +1,8 @@ +# DataFog Roadmap + +The roadmap — current release status (4.7.x) and v5.0.0 plans — lives in +[docs/roadmap.rst](docs/roadmap.rst). + +Roadmap priorities are shaped by user feedback — open a +[GitHub issue](https://github.com/DataFog/datafog-python/issues) or join +the [Discord](https://discord.gg/bzDth394R4). diff --git a/benchmarks/README.md b/benchmarks/README.md new file mode 100644 index 00000000..93ec4923 --- /dev/null +++ b/benchmarks/README.md @@ -0,0 +1,151 @@ +# DataFog benchmarks + +Every performance number DataFog publishes should be reproducible by a +skeptic with one command. This directory is that command: + +```bash +pip install -e . -r benchmarks/requirements.txt +python -m spacy download en_core_web_sm # comparison suites only +python benchmarks/run.py +``` + +Suites that are missing optional dependencies are skipped with an install +hint — `python benchmarks/run.py --suite core,hook` runs with nothing but +datafog itself installed. `--json results.json` writes machine-readable +output; `--quick` is a fast smoke run (fewer repeats, large payloads +skipped). + +## Methodology + +- **In-process timings** use stdlib `timeit`: `Timer.autorange()` picks an + inner loop count so each timed repeat runs ≥0.2s, then 5 repeats (3 in + comparison suites) are reported as median ± stdev. No mocking — every + call goes through the same public code path a user hits. +- **The hook suite measures wall-clock subprocess time**: a fresh + `datafog-hook` process per iteration, JSON payload on stdin, exactly as + Claude Code invokes it. 3 warmup runs, 30 timed runs, median and p90 + reported. Python interpreter startup is _included_ — that is the honest + cost of a per-tool-call hook. +- **Pinned payloads** live in [payloads/](payloads/) and are checked in, + not generated at runtime. [payloads/manifest.json](payloads/manifest.json) + records the entity counts the regex engine must find in each payload; the + runner verifies those counts _before_ timing anything, so the suite fails + loudly rather than silently benchmarking an engine that stopped + detecting. All PII values are industry-standard synthetic test data + (reserved `.invalid`/`example.com` domains, fictional 555-01XX phone + numbers, public Luhn-valid test cards, SSA example SSNs). + +| Payload | Size | Content | +| ------------------------- | ------ | -------------------------------------------------------------------------- | +| `core_1kb_dense.txt` | 1.2 KB | support ticket, 8 entities (PII-dense) | +| `core_10kb_mixed.txt` | 10 KB | business document, 12 entities in prose | +| `core_100kb_sparse.txt` | 100 KB | machine logs, 18 entities among UUIDs/timestamps (false-positive pressure) | +| `chat_request_clean.json` | — | 2-message LiteLLM chat body, no PII | +| `chat_request_pii.json` | — | 2-message LiteLLM chat body, email + card + phone | +| `hook_pretooluse.json` | — | Claude Code `PreToolUse` payload, `curl` command carrying PII | + +All suites use the production default entity set +(`EMAIL, PHONE, CREDIT_CARD, SSN`). + +## Reference results + +Apple M5 Pro, macOS, CPython 3.13, datafog 4.7.0, litellm 1.91.0, +presidio-analyzer 2.2.363, spaCy 3.8 (`en_core_web_sm`). Medians of the +full (non-quick) run; expect different absolute numbers on different +hardware, but the ratios and orders of magnitude should hold. + +### core — `datafog.scan` / `datafog.redact`, regex engine + +| Operation | Median | Throughput | +| -------------------- | ------- | ---------- | +| scan 1.2 KB dense | 231 µs | 5.3 MB/s | +| redact 1.2 KB dense | 240 µs | 5.1 MB/s | +| scan 10 KB mixed | 1.20 ms | 8.6 MB/s | +| redact 10 KB mixed | 1.19 ms | 8.6 MB/s | +| scan 100 KB sparse | 11.9 ms | 8.4 MB/s | +| redact 100 KB sparse | 12.1 ms | 8.3 MB/s | + +### guardrail — `DataFogGuardrail` (LiteLLM), in-process + +| Operation | Median | +| ------------------------------------------------------ | ------ | +| single message redact | 43 µs | +| `pre_call`, clean 2-message request | 118 µs | +| `pre_call`, PII 2-message request (redact) | 222 µs | +| event-loop dispatch (harness overhead, included above) | 27 µs | + +The PII-request figure includes litellm's own guardrail-intervention +logging (~85 µs), not just detection. In a live proxy the event loop is +already running, so the ~27 µs dispatch overhead in the two per-request +rows is an artifact of benchmarking with `run_until_complete`. + +### hook — `datafog-hook` (Claude Code) + +| Operation | Median | +| ------------------------------------------- | ------------------------------------ | +| end-to-end subprocess, incl. Python startup | 69–89 ms across runs (p90 ~77–96 ms) | +| the scan itself (in-process) | 75 µs | + +The end-to-end cost is ~99.9% Python interpreter + import startup; the +scan is microseconds. Cold (first-ever) invocations can spike to several +hundred ms while the OS warms caches — the warmup runs exclude that, the +p90 shows steady-state spread. + +### Comparisons — same payloads, same four entity types + +Both comparison targets are pinned to `en_core_web_sm`, the _smallest_ +English spaCy model — deliberately favorable to them (smaller model = +faster inference), so these ratios are lower bounds. Model/engine setup +time (~0.1–1 s) is excluded from per-scan figures. Presidio runs in-process +via `AnalyzerEngine` with a documented `NlpEngineProvider` config — no +sidecar, no network hop, which again flatters Presidio relative to its +usual proxy deployment. + +| Payload | datafog regex | Presidio | spaCy NER | datafog vs Presidio | datafog vs spaCy | +| ------------- | ------------- | -------- | --------- | ------------------- | ---------------- | +| 1.2 KB dense | 222–251 µs | 22.8 ms | 28.6 ms | 103x | 114x | +| 10 KB mixed | 1.2 ms | 174 ms | 171 ms | 148x | 140x | +| 100 KB sparse | 11.9–12.2 ms | 2.02 s | 1.60 s | 170x | 131x | + +Speed is not the only axis: the runner records what each engine actually +detected on each payload (see the `detected` metadata in the output). +Recall differs in both directions — e.g. Presidio's email recognizer does +not match addresses on the reserved `.invalid` domain, and context-based +scoring drops some entities the regex engine finds; conversely NER models +detect names and locations the regex engine does not target at all. The +comparison is a fair _speed_ comparison on identical inputs and entity +types, not an accuracy study. + +## How these map to published claims + +- **"~31µs per request" (LiteLLM guardrail, project README / + datafog.ai)** — not reproduced _as worded_. ~31–43 µs is the cost of + scanning/redacting **one short message**; a full 2-message request + through `async_pre_call_hook` is ~120 µs clean / ~220 µs with redaction + and litellm logging. Still 100–1000x below a sidecar's network hop, but + the claim should be reworded to per-message, e.g. _"~40µs per message — + a request clears the guardrail in well under a millisecond"_. Reproduce: + `python benchmarks/run.py --suite guardrail`. +- **"~70ms per invocation including process startup" (Claude Code + hook)** — reproduced: 69–89 ms median across runs on this machine + (~70 ms idle, higher under load), dominated by interpreter startup. + _"~70–90ms"_ (or "under 100ms") is the defensible phrasing. Reproduce: `python benchmarks/run.py --suite hook`. +- **"190x performance advantage" (PyPI tagline)** — not reproduced at + 190x by this suite. Measured: 103–170x vs Presidio and 114–140x vs spaCy + NER, varying by payload. The historical 190x came from a different + (13.3 KB, entity-dense) document and engine configuration. Honest + phrasings this suite supports: _"100x+ faster than NER-based PII + detection"_ or _"up to 170x faster than Presidio on identical + payloads"_. Reproduce: `python benchmarks/run.py --suite spacy,presidio`. + +## Fairness notes / known limitations + +- Single-threaded, single-machine, synthetic payloads. Real chat traffic + and real documents will differ; the payloads are designed to bracket the + realistic range (dense, mixed, sparse). +- `timeit` medians hide GC pauses and tail latency by design; the hook + suite's p90 is the only tail-latency figure here. +- The comparison suites measure the engines _as configured here_ + (documented model, in-process, setup excluded). Different Presidio + recognizer registries or larger spaCy models change both speed and + recall — in Presidio's favor on recall, against it on speed. diff --git a/benchmarks/_suites.py b/benchmarks/_suites.py new file mode 100644 index 00000000..9c772faa --- /dev/null +++ b/benchmarks/_suites.py @@ -0,0 +1,379 @@ +"""Benchmark suite implementations. + +Each ``suite_*`` function returns a list of Measurements. Suites verify +their expected detections against ``payloads/manifest.json`` *before* +timing anything, so the suite cannot silently benchmark an engine that +stopped detecting. + +Suites raise :class:`SuiteUnavailable` when an optional dependency is +missing; the runner reports the skip and the install hint. +""" + +from __future__ import annotations + +import asyncio +import dataclasses +import json +import shutil +import subprocess +import sys +import time +from pathlib import Path +from typing import Any, Callable + +from _timing import Measurement, bench, bench_wall + +PAYLOAD_DIR = Path(__file__).parent / "payloads" + +# The comparison suites pin the *small* English model. This is deliberately +# favorable to spaCy/Presidio (smaller model = faster inference), so the +# measured speed ratios are lower bounds, not cherry-picked upper bounds. +COMPARISON_SPACY_MODEL = "en_core_web_sm" + +# Presidio's names for the same four entity types the datafog suites use. +PRESIDIO_ENTITIES = ["EMAIL_ADDRESS", "PHONE_NUMBER", "CREDIT_CARD", "US_SSN"] + + +class SuiteUnavailable(Exception): + """Raised when a suite's optional dependencies are not installed.""" + + +def load_manifest() -> dict[str, Any]: + return json.loads((PAYLOAD_DIR / "manifest.json").read_text()) + + +def _scan_counts(text: str, entity_types: list[str]) -> dict[str, int]: + import datafog + + counts: dict[str, int] = {} + for entity in datafog.scan( + text, engine="regex", entity_types=entity_types + ).entities: + counts[entity.type] = counts.get(entity.type, 0) + 1 + return counts + + +def _verify_counts( + text: str, entity_types: list[str], expected: dict[str, int], name: str +) -> None: + actual = _scan_counts(text, entity_types) + if actual != expected: + raise RuntimeError( + f"payload {name}: detected entity counts {actual} do not match " + f"manifest {expected} — refusing to benchmark unverified detection" + ) + + +def _core_payloads() -> list[tuple[str, str, dict[str, Any]]]: + manifest = load_manifest() + entity_types = manifest["entity_types"] + payloads = [] + for name, spec in sorted( + manifest["payloads"].items(), key=lambda kv: kv[1]["bytes"] + ): + text = (PAYLOAD_DIR / name).read_text() + _verify_counts(text, entity_types, spec["expected_counts"], name) + payloads.append((name, text, spec)) + return payloads + + +def _throughput(size_bytes: int, median_us: float) -> float: + # bytes / microsecond is numerically identical to MB/s (1e6 / 1e6). + return size_bytes / median_us + + +def suite_core(quick: bool) -> list[Measurement]: + """datafog.scan / datafog.redact with the regex engine on pinned payloads.""" + import datafog + + entity_types = load_manifest()["entity_types"] + repeats = 3 if quick else 5 + results = [] + for name, text, spec in _core_payloads(): + for op, func in ( + ( + "scan", + lambda t=text: datafog.scan( + t, engine="regex", entity_types=entity_types + ), + ), + ( + "redact", + lambda t=text: datafog.redact( + t, engine="regex", entity_types=entity_types + ), + ), + ): + m = bench( + "core", + f"{op} {name}", + func, + repeats=repeats, + meta={ + "bytes": spec["bytes"], + "entities": sum(spec["expected_counts"].values()), + }, + ) + throughput = round(_throughput(spec["bytes"], m.median), 1) + results.append( + dataclasses.replace(m, meta={**m.meta, "throughput_mb_s": throughput}) + ) + return results + + +def suite_guardrail(quick: bool) -> list[Measurement]: + """DataFogGuardrail (LiteLLM) per-request latency, in-process.""" + try: + from datafog.integrations.litellm_guardrail import DataFogGuardrail + except ImportError as exc: + raise SuiteUnavailable( + f"litellm/fastapi not installed ({exc.name}) — " + "pip install -r benchmarks/requirements.txt" + ) from exc + import datafog + + clean = json.loads((PAYLOAD_DIR / "chat_request_clean.json").read_text()) + pii = json.loads((PAYLOAD_DIR / "chat_request_pii.json").read_text()) + guardrail = DataFogGuardrail(action="redact") + loop = asyncio.new_event_loop() + try: + # Verify behavior before timing: PII request must come back redacted, + # clean request must pass through untouched. + redacted = loop.run_until_complete( + guardrail.async_pre_call_hook(None, None, pii, "completion") + ) + if "[EMAIL_" not in redacted["messages"][1]["content"]: + raise RuntimeError("guardrail did not redact the PII chat request") + passthrough = loop.run_until_complete( + guardrail.async_pre_call_hook(None, None, clean, "completion") + ) + if passthrough is not clean: + raise RuntimeError("guardrail modified a clean chat request") + + repeats = 3 if quick else 5 + + async def _noop() -> None: + pass + + def timed(data: dict) -> Callable[[], Any]: + return lambda: loop.run_until_complete( + guardrail.async_pre_call_hook(None, None, data, "completion") + ) + + pii_msg = pii["messages"][1]["content"] + entity_types = guardrail.entity_types + results = [ + bench( + "guardrail", + "event-loop dispatch (harness overhead)", + lambda: loop.run_until_complete(_noop()), + repeats=repeats, + meta={ + "note": "subtract from per-request figures; a live proxy's loop is already running" + }, + ), + bench( + "guardrail", + "pre_call, clean 2-message request", + timed(clean), + repeats=repeats, + meta={ + "note": "the common case: scan finds nothing, request passes through" + }, + ), + bench( + "guardrail", + "pre_call, PII 2-message request (redact)", + timed(pii), + repeats=repeats, + meta={ + "note": "includes litellm's guardrail-intervention logging record" + }, + ), + bench( + "guardrail", + "single message redact (datafog.redact)", + lambda: datafog.redact( + pii_msg, engine="regex", entity_types=entity_types + ), + repeats=repeats, + meta={"bytes": len(pii_msg.encode())}, + ), + ] + finally: + loop.close() + return results + + +def _hook_command() -> list[str]: + exe = shutil.which("datafog-hook") + if exe: + return [exe] + return [sys.executable, "-m", "datafog.integrations.claude_code"] + + +def suite_hook(quick: bool) -> list[Measurement]: + """datafog-hook end-to-end (fresh process per call, as Claude Code runs it).""" + payload = (PAYLOAD_DIR / "hook_pretooluse.json").read_text() + cmd = _hook_command() + + def run_once() -> subprocess.CompletedProcess: + return subprocess.run(cmd, input=payload, capture_output=True, text=True) + + # Verify before timing: the hook must actually fire on this payload. + proc = run_once() + if ( + proc.returncode != 0 + or "PreToolUse" not in proc.stdout + or "EMAIL" not in proc.stdout + ): + raise RuntimeError( + f"datafog-hook did not flag the pinned payload " + f"(exit {proc.returncode}, stdout {proc.stdout!r}, stderr {proc.stderr!r})" + ) + + from datafog.integrations.claude_code import run as hook_run + + payload_obj = json.loads(payload) + return [ + bench_wall( + "hook", + "datafog-hook end-to-end (subprocess, incl. Python startup)", + run_once, + iterations=10 if quick else 30, + meta={"command": " ".join(cmd)}, + ), + bench( + "hook", + "hook scan only (in-process run())", + lambda: hook_run(payload_obj, {}), + repeats=3 if quick else 5, + meta={ + "note": "the scan itself; the rest of the end-to-end figure is process startup" + }, + ), + ] + + +def suite_spacy(quick: bool) -> list[Measurement]: + """datafog regex engine vs the spaCy NER pipeline on the same payloads.""" + try: + from datafog.processing.text_processing.spacy_pii_annotator import ( + SpacyPIIAnnotator, + ) + + annotator = SpacyPIIAnnotator.create(model_name=COMPARISON_SPACY_MODEL) + except Exception as exc: + raise SuiteUnavailable( + f"spaCy model unavailable ({exc}) — pip install -r benchmarks/requirements.txt " + f"&& python -m spacy download {COMPARISON_SPACY_MODEL}" + ) from exc + import datafog + + entity_types = load_manifest()["entity_types"] + results = [] + for name, text, spec in _core_payloads(): + if quick and spec["bytes"] > 20_000: + continue + regex_m = bench( + "spacy", + f"regex scan {name}", + lambda t=text: datafog.scan(t, engine="regex", entity_types=entity_types), + repeats=3, + meta={"bytes": spec["bytes"]}, + ) + spacy_m = bench( + "spacy", + f"spaCy NER ({COMPARISON_SPACY_MODEL}) {name}", + lambda t=text: annotator.annotate(t), + repeats=3, + meta={ + "bytes": spec["bytes"], + "note": "model load excluded; different entity classes than regex", + }, + ) + speedup = round(spacy_m.median / regex_m.median, 1) + spacy_m = dataclasses.replace( + spacy_m, meta={**spacy_m.meta, "regex_speedup_x": speedup} + ) + results.extend([regex_m, spacy_m]) + return results + + +def suite_presidio(quick: bool) -> list[Measurement]: + """presidio-analyzer on the same payloads, same four entity types.""" + try: + from presidio_analyzer import AnalyzerEngine + from presidio_analyzer.nlp_engine import NlpEngineProvider + except ImportError as exc: + raise SuiteUnavailable( + f"presidio-analyzer not installed ({exc.name}) — " + "pip install -r benchmarks/requirements.txt" + ) from exc + import datafog + + nlp_config = { + "nlp_engine_name": "spacy", + "models": [{"lang_code": "en", "model_name": COMPARISON_SPACY_MODEL}], + } + start = time.perf_counter() + engine = AnalyzerEngine( + nlp_engine=NlpEngineProvider(nlp_configuration=nlp_config).create_engine() + ) + setup_ms = (time.perf_counter() - start) * 1e3 + + entity_types = load_manifest()["entity_types"] + results = [ + Measurement( + suite="presidio", + name=f"AnalyzerEngine setup ({COMPARISON_SPACY_MODEL})", + unit="ms/run", + median=setup_ms, + mean=setup_ms, + stdev=0.0, + best=setup_ms, + inner_loops=1, + repeats=1, + meta={"note": "one-time cost, excluded from per-scan figures below"}, + ) + ] + for name, text, spec in _core_payloads(): + if quick and spec["bytes"] > 20_000: + continue + datafog_m = bench( + "presidio", + f"datafog regex scan {name}", + lambda t=text: datafog.scan(t, engine="regex", entity_types=entity_types), + repeats=3, + meta={"bytes": spec["bytes"], "detected": _scan_counts(text, entity_types)}, + ) + presidio_counts: dict[str, int] = {} + for r in engine.analyze(text=text, entities=PRESIDIO_ENTITIES, language="en"): + presidio_counts[r.entity_type] = presidio_counts.get(r.entity_type, 0) + 1 + presidio_m = bench( + "presidio", + f"presidio analyze {name}", + lambda t=text: engine.analyze( + text=t, entities=PRESIDIO_ENTITIES, language="en" + ), + repeats=3, + meta={ + "bytes": spec["bytes"], + "detected": dict(sorted(presidio_counts.items())), + }, + ) + speedup = round(presidio_m.median / datafog_m.median, 1) + presidio_m = dataclasses.replace( + presidio_m, meta={**presidio_m.meta, "datafog_speedup_x": speedup} + ) + results.extend([datafog_m, presidio_m]) + return results + + +SUITES: dict[str, Callable[[bool], list[Measurement]]] = { + "core": suite_core, + "guardrail": suite_guardrail, + "hook": suite_hook, + "spacy": suite_spacy, + "presidio": suite_presidio, +} diff --git a/benchmarks/_timing.py b/benchmarks/_timing.py new file mode 100644 index 00000000..44d878c4 --- /dev/null +++ b/benchmarks/_timing.py @@ -0,0 +1,98 @@ +"""Measurement harness for the benchmark suite. + +Thin wrapper over :mod:`timeit` for in-process microbenchmarks, plus a +wall-clock harness for whole-process measurements (the Claude Code hook). +Every measurement reports the spread across repeats, not a single number, +so noisy runs are visible instead of hidden. +""" + +from __future__ import annotations + +import statistics +import time +from dataclasses import asdict, dataclass, field +from timeit import Timer +from typing import Any, Callable + + +@dataclass(frozen=True) +class Measurement: + """One benchmarked operation, with per-op stats across repeats.""" + + suite: str + name: str + unit: str # "us/op" or "ms/run" + median: float + mean: float + stdev: float + best: float + inner_loops: int # ops per timed repeat (1 for wall-clock runs) + repeats: int + meta: dict[str, Any] = field(default_factory=dict) + + def to_dict(self) -> dict[str, Any]: + return asdict(self) + + +def _stats( + suite: str, + name: str, + unit: str, + samples: list[float], + inner_loops: int, + meta: dict[str, Any] | None, +) -> Measurement: + return Measurement( + suite=suite, + name=name, + unit=unit, + median=statistics.median(samples), + mean=statistics.fmean(samples), + stdev=statistics.stdev(samples) if len(samples) > 1 else 0.0, + best=min(samples), + inner_loops=inner_loops, + repeats=len(samples), + meta=dict(meta or {}), + ) + + +def bench( + suite: str, + name: str, + func: Callable[[], Any], + repeats: int = 5, + meta: dict[str, Any] | None = None, +) -> Measurement: + """Time ``func`` in-process; per-op microseconds across ``repeats``. + + Uses ``Timer.autorange`` to pick an inner loop count large enough for + each timed repeat to run at least ~0.2s, so per-op figures are not + dominated by timer resolution. + """ + timer = Timer(func) + inner_loops, _ = timer.autorange() + per_op_us = [ + total / inner_loops * 1e6 + for total in timer.repeat(repeat=repeats, number=inner_loops) + ] + return _stats(suite, name, "us/op", per_op_us, inner_loops, meta) + + +def bench_wall( + suite: str, + name: str, + run_once: Callable[[], Any], + iterations: int = 30, + warmup: int = 3, + meta: dict[str, Any] | None = None, +) -> Measurement: + """Wall-clock time ``run_once`` (e.g. a subprocess); milliseconds per run.""" + for _ in range(warmup): + run_once() + samples_ms: list[float] = [] + for _ in range(iterations): + start = time.perf_counter() + run_once() + samples_ms.append((time.perf_counter() - start) * 1e3) + extra = {"p90": statistics.quantiles(samples_ms, n=10)[-1], **(meta or {})} + return _stats(suite, name, "ms/run", samples_ms, 1, extra) diff --git a/benchmarks/payloads/chat_request_clean.json b/benchmarks/payloads/chat_request_clean.json new file mode 100644 index 00000000..a0a3fb83 --- /dev/null +++ b/benchmarks/payloads/chat_request_clean.json @@ -0,0 +1,13 @@ +{ + "model": "gpt-4o-mini", + "messages": [ + { + "role": "system", + "content": "You are a concise assistant for a customer support team. Answer in plain language and never invent account details." + }, + { + "role": "user", + "content": "Summarize the attached quarterly report in three bullet points and suggest one follow-up question for the finance team." + } + ] +} diff --git a/benchmarks/payloads/chat_request_pii.json b/benchmarks/payloads/chat_request_pii.json new file mode 100644 index 00000000..1c5471b4 --- /dev/null +++ b/benchmarks/payloads/chat_request_pii.json @@ -0,0 +1,13 @@ +{ + "model": "gpt-4o-mini", + "messages": [ + { + "role": "system", + "content": "You are a concise assistant for a customer support team. Answer in plain language and never invent account details." + }, + { + "role": "user", + "content": "Draft a follow-up email to alex.rivera@example.invalid, mention the declined card 4242424242424242, and ask them to call (212) 555-0142 to confirm." + } + ] +} diff --git a/benchmarks/payloads/core_100kb_sparse.txt b/benchmarks/payloads/core_100kb_sparse.txt new file mode 100644 index 00000000..6c22952c --- /dev/null +++ b/benchmarks/payloads/core_100kb_sparse.txt @@ -0,0 +1,1171 @@ +2026-07-01T08:00:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000000 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:01Z DEBUG cache.miss key=report:1 shard=3 ttl=900 +2026-07-01T08:00:02Z INFO worker.job id=2 queue=exports attempt=1 result=ok +2026-07-01T08:00:03Z WARN rate.limit tenant=t_3 remaining=17 window=60s +2026-07-01T08:00:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000004 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:05Z DEBUG cache.miss key=report:5 shard=3 ttl=900 +2026-07-01T08:00:06Z INFO worker.job id=6 queue=exports attempt=1 result=ok +2026-07-01T08:00:07Z WARN rate.limit tenant=t_7 remaining=17 window=60s +2026-07-01T08:00:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000008 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:09Z DEBUG cache.miss key=report:9 shard=3 ttl=900 +2026-07-01T08:00:10Z INFO worker.job id=10 queue=exports attempt=1 result=ok +2026-07-01T08:00:11Z WARN rate.limit tenant=t_11 remaining=17 window=60s +2026-07-01T08:00:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000012 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:13Z DEBUG cache.miss key=report:13 shard=3 ttl=900 +2026-07-01T08:00:14Z INFO worker.job id=14 queue=exports attempt=1 result=ok +2026-07-01T08:00:15Z WARN rate.limit tenant=t_15 remaining=17 window=60s +2026-07-01T08:00:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000016 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:17Z DEBUG cache.miss key=report:17 shard=3 ttl=900 +2026-07-01T08:00:18Z INFO worker.job id=18 queue=exports attempt=1 result=ok +2026-07-01T08:00:19Z WARN rate.limit tenant=t_19 remaining=17 window=60s +2026-07-01T08:00:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000020 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:21Z DEBUG cache.miss key=report:21 shard=3 ttl=900 +2026-07-01T08:00:22Z INFO worker.job id=22 queue=exports attempt=1 result=ok +2026-07-01T08:00:23Z WARN rate.limit tenant=t_23 remaining=17 window=60s +2026-07-01T08:00:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000024 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:25Z DEBUG cache.miss key=report:25 shard=3 ttl=900 +2026-07-01T08:00:26Z INFO worker.job id=26 queue=exports attempt=1 result=ok +2026-07-01T08:00:27Z WARN rate.limit tenant=t_27 remaining=17 window=60s +2026-07-01T08:00:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000028 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:29Z DEBUG cache.miss key=report:29 shard=3 ttl=900 +2026-07-01T08:00:30Z INFO worker.job id=30 queue=exports attempt=1 result=ok +2026-07-01T08:00:31Z WARN rate.limit tenant=t_31 remaining=17 window=60s +2026-07-01T08:00:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000032 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:33Z DEBUG cache.miss key=report:33 shard=3 ttl=900 +2026-07-01T08:00:34Z INFO worker.job id=34 queue=exports attempt=1 result=ok +2026-07-01T08:00:35Z WARN rate.limit tenant=t_35 remaining=17 window=60s +2026-07-01T08:00:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000036 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:37Z DEBUG cache.miss key=report:37 shard=3 ttl=900 +2026-07-01T08:00:38Z INFO worker.job id=38 queue=exports attempt=1 result=ok +2026-07-01T08:00:39Z WARN rate.limit tenant=t_39 remaining=17 window=60s +2026-07-01T08:00:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000040 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:41Z DEBUG cache.miss key=report:41 shard=3 ttl=900 +2026-07-01T08:00:42Z INFO worker.job id=42 queue=exports attempt=1 result=ok +2026-07-01T08:00:43Z WARN rate.limit tenant=t_43 remaining=17 window=60s +2026-07-01T08:00:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000044 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:45Z DEBUG cache.miss key=report:45 shard=3 ttl=900 +2026-07-01T08:00:46Z INFO worker.job id=46 queue=exports attempt=1 result=ok +2026-07-01T08:00:47Z WARN rate.limit tenant=t_47 remaining=17 window=60s +2026-07-01T08:00:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000048 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:49Z DEBUG cache.miss key=report:49 shard=3 ttl=900 +2026-07-01T08:00:50Z INFO worker.job id=50 queue=exports attempt=1 result=ok +2026-07-01T08:00:51Z WARN rate.limit tenant=t_51 remaining=17 window=60s +2026-07-01T08:00:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000052 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:53Z DEBUG cache.miss key=report:53 shard=3 ttl=900 +2026-07-01T08:00:54Z INFO worker.job id=54 queue=exports attempt=1 result=ok +2026-07-01T08:00:55Z WARN rate.limit tenant=t_55 remaining=17 window=60s +2026-07-01T08:00:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000056 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:00:57Z DEBUG cache.miss key=report:57 shard=3 ttl=900 +2026-07-01T08:00:58Z INFO worker.job id=58 queue=exports attempt=1 result=ok +2026-07-01T08:00:59Z WARN rate.limit tenant=t_59 remaining=17 window=60s +2026-07-01T08:01:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000060 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:01Z DEBUG cache.miss key=report:61 shard=3 ttl=900 +2026-07-01T08:01:02Z INFO worker.job id=62 queue=exports attempt=1 result=ok +2026-07-01T08:01:03Z WARN rate.limit tenant=t_63 remaining=17 window=60s +2026-07-01T08:01:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000064 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:05Z DEBUG cache.miss key=report:65 shard=3 ttl=900 +2026-07-01T08:01:06Z INFO worker.job id=66 queue=exports attempt=1 result=ok +2026-07-01T08:01:07Z WARN rate.limit tenant=t_67 remaining=17 window=60s +2026-07-01T08:01:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000068 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:09Z DEBUG cache.miss key=report:69 shard=3 ttl=900 +2026-07-01T08:01:10Z INFO worker.job id=70 queue=exports attempt=1 result=ok +2026-07-01T08:01:11Z WARN rate.limit tenant=t_71 remaining=17 window=60s +2026-07-01T08:01:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000072 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:13Z DEBUG cache.miss key=report:73 shard=3 ttl=900 +2026-07-01T08:01:14Z INFO worker.job id=74 queue=exports attempt=1 result=ok +2026-07-01T08:01:15Z WARN rate.limit tenant=t_75 remaining=17 window=60s +2026-07-01T08:01:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000076 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:17Z DEBUG cache.miss key=report:77 shard=3 ttl=900 +2026-07-01T08:01:18Z INFO worker.job id=78 queue=exports attempt=1 result=ok +2026-07-01T08:01:19Z WARN rate.limit tenant=t_79 remaining=17 window=60s +2026-07-01T08:01:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000080 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:21Z DEBUG cache.miss key=report:81 shard=3 ttl=900 +2026-07-01T08:01:22Z INFO worker.job id=82 queue=exports attempt=1 result=ok +2026-07-01T08:01:23Z WARN rate.limit tenant=t_83 remaining=17 window=60s +2026-07-01T08:01:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000084 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:25Z DEBUG cache.miss key=report:85 shard=3 ttl=900 +2026-07-01T08:01:26Z INFO worker.job id=86 queue=exports attempt=1 result=ok +2026-07-01T08:01:27Z WARN rate.limit tenant=t_87 remaining=17 window=60s +2026-07-01T08:01:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000088 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:29Z DEBUG cache.miss key=report:89 shard=3 ttl=900 +2026-07-01T08:01:30Z INFO worker.job id=90 queue=exports attempt=1 result=ok +2026-07-01T08:01:31Z WARN rate.limit tenant=t_91 remaining=17 window=60s +2026-07-01T08:01:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000092 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:33Z DEBUG cache.miss key=report:93 shard=3 ttl=900 +2026-07-01T08:01:34Z INFO worker.job id=94 queue=exports attempt=1 result=ok +2026-07-01T08:01:35Z WARN rate.limit tenant=t_95 remaining=17 window=60s +2026-07-01T08:01:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000096 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:37Z DEBUG cache.miss key=report:97 shard=3 ttl=900 +2026-07-01T08:01:38Z INFO worker.job id=98 queue=exports attempt=1 result=ok +2026-07-01T08:01:39Z WARN rate.limit tenant=t_99 remaining=17 window=60s +2026-07-01T08:01:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000100 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:41Z DEBUG cache.miss key=report:101 shard=3 ttl=900 +2026-07-01T08:01:42Z INFO worker.job id=102 queue=exports attempt=1 result=ok +2026-07-01T08:01:43Z WARN rate.limit tenant=t_103 remaining=17 window=60s +2026-07-01T08:01:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000104 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:45Z DEBUG cache.miss key=report:105 shard=3 ttl=900 +2026-07-01T08:01:46Z INFO worker.job id=106 queue=exports attempt=1 result=ok +2026-07-01T08:01:47Z WARN rate.limit tenant=t_107 remaining=17 window=60s +2026-07-01T08:01:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000108 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:49Z DEBUG cache.miss key=report:109 shard=3 ttl=900 +2026-07-01T08:01:50Z INFO worker.job id=110 queue=exports attempt=1 result=ok +2026-07-01T08:01:51Z WARN rate.limit tenant=t_111 remaining=17 window=60s +2026-07-01T08:01:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000112 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:53Z DEBUG cache.miss key=report:113 shard=3 ttl=900 +2026-07-01T08:01:54Z INFO worker.job id=114 queue=exports attempt=1 result=ok +2026-07-01T08:01:55Z WARN rate.limit tenant=t_115 remaining=17 window=60s +2026-07-01T08:01:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000116 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:01:57Z DEBUG cache.miss key=report:117 shard=3 ttl=900 +2026-07-01T08:01:58Z INFO worker.job id=118 queue=exports attempt=1 result=ok +2026-07-01T08:01:59Z WARN rate.limit tenant=t_119 remaining=17 window=60s +2026-07-01T08:02:00Z WARN pii.echo user_email=j.chen@support.example.com callback=415-555-0188 +2026-07-01T08:02:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000120 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:01Z DEBUG cache.miss key=report:121 shard=3 ttl=900 +2026-07-01T08:02:02Z INFO worker.job id=122 queue=exports attempt=1 result=ok +2026-07-01T08:02:03Z WARN rate.limit tenant=t_123 remaining=17 window=60s +2026-07-01T08:02:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000124 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:05Z DEBUG cache.miss key=report:125 shard=3 ttl=900 +2026-07-01T08:02:06Z INFO worker.job id=126 queue=exports attempt=1 result=ok +2026-07-01T08:02:07Z WARN rate.limit tenant=t_127 remaining=17 window=60s +2026-07-01T08:02:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000128 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:09Z DEBUG cache.miss key=report:129 shard=3 ttl=900 +2026-07-01T08:02:10Z INFO worker.job id=130 queue=exports attempt=1 result=ok +2026-07-01T08:02:11Z WARN rate.limit tenant=t_131 remaining=17 window=60s +2026-07-01T08:02:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000132 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:13Z DEBUG cache.miss key=report:133 shard=3 ttl=900 +2026-07-01T08:02:14Z INFO worker.job id=134 queue=exports attempt=1 result=ok +2026-07-01T08:02:15Z WARN rate.limit tenant=t_135 remaining=17 window=60s +2026-07-01T08:02:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000136 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:17Z DEBUG cache.miss key=report:137 shard=3 ttl=900 +2026-07-01T08:02:18Z INFO worker.job id=138 queue=exports attempt=1 result=ok +2026-07-01T08:02:19Z WARN rate.limit tenant=t_139 remaining=17 window=60s +2026-07-01T08:02:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000140 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:21Z DEBUG cache.miss key=report:141 shard=3 ttl=900 +2026-07-01T08:02:22Z INFO worker.job id=142 queue=exports attempt=1 result=ok +2026-07-01T08:02:23Z WARN rate.limit tenant=t_143 remaining=17 window=60s +2026-07-01T08:02:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000144 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:25Z DEBUG cache.miss key=report:145 shard=3 ttl=900 +2026-07-01T08:02:26Z INFO worker.job id=146 queue=exports attempt=1 result=ok +2026-07-01T08:02:27Z WARN rate.limit tenant=t_147 remaining=17 window=60s +2026-07-01T08:02:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000148 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:29Z DEBUG cache.miss key=report:149 shard=3 ttl=900 +2026-07-01T08:02:30Z INFO worker.job id=150 queue=exports attempt=1 result=ok +2026-07-01T08:02:31Z WARN rate.limit tenant=t_151 remaining=17 window=60s +2026-07-01T08:02:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000152 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:33Z DEBUG cache.miss key=report:153 shard=3 ttl=900 +2026-07-01T08:02:34Z INFO worker.job id=154 queue=exports attempt=1 result=ok +2026-07-01T08:02:35Z WARN rate.limit tenant=t_155 remaining=17 window=60s +2026-07-01T08:02:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000156 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:37Z DEBUG cache.miss key=report:157 shard=3 ttl=900 +2026-07-01T08:02:38Z INFO worker.job id=158 queue=exports attempt=1 result=ok +2026-07-01T08:02:39Z WARN rate.limit tenant=t_159 remaining=17 window=60s +2026-07-01T08:02:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000160 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:41Z DEBUG cache.miss key=report:161 shard=3 ttl=900 +2026-07-01T08:02:42Z INFO worker.job id=162 queue=exports attempt=1 result=ok +2026-07-01T08:02:43Z WARN rate.limit tenant=t_163 remaining=17 window=60s +2026-07-01T08:02:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000164 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:45Z DEBUG cache.miss key=report:165 shard=3 ttl=900 +2026-07-01T08:02:46Z INFO worker.job id=166 queue=exports attempt=1 result=ok +2026-07-01T08:02:47Z WARN rate.limit tenant=t_167 remaining=17 window=60s +2026-07-01T08:02:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000168 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:49Z DEBUG cache.miss key=report:169 shard=3 ttl=900 +2026-07-01T08:02:50Z INFO worker.job id=170 queue=exports attempt=1 result=ok +2026-07-01T08:02:51Z WARN rate.limit tenant=t_171 remaining=17 window=60s +2026-07-01T08:02:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000172 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:53Z DEBUG cache.miss key=report:173 shard=3 ttl=900 +2026-07-01T08:02:54Z INFO worker.job id=174 queue=exports attempt=1 result=ok +2026-07-01T08:02:55Z WARN rate.limit tenant=t_175 remaining=17 window=60s +2026-07-01T08:02:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000176 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:02:57Z DEBUG cache.miss key=report:177 shard=3 ttl=900 +2026-07-01T08:02:58Z INFO worker.job id=178 queue=exports attempt=1 result=ok +2026-07-01T08:02:59Z WARN rate.limit tenant=t_179 remaining=17 window=60s +2026-07-01T08:03:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000180 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:01Z DEBUG cache.miss key=report:181 shard=3 ttl=900 +2026-07-01T08:03:02Z INFO worker.job id=182 queue=exports attempt=1 result=ok +2026-07-01T08:03:03Z WARN rate.limit tenant=t_183 remaining=17 window=60s +2026-07-01T08:03:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000184 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:05Z DEBUG cache.miss key=report:185 shard=3 ttl=900 +2026-07-01T08:03:06Z INFO worker.job id=186 queue=exports attempt=1 result=ok +2026-07-01T08:03:07Z WARN rate.limit tenant=t_187 remaining=17 window=60s +2026-07-01T08:03:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000188 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:09Z DEBUG cache.miss key=report:189 shard=3 ttl=900 +2026-07-01T08:03:10Z INFO worker.job id=190 queue=exports attempt=1 result=ok +2026-07-01T08:03:11Z WARN rate.limit tenant=t_191 remaining=17 window=60s +2026-07-01T08:03:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000192 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:13Z DEBUG cache.miss key=report:193 shard=3 ttl=900 +2026-07-01T08:03:14Z INFO worker.job id=194 queue=exports attempt=1 result=ok +2026-07-01T08:03:15Z WARN rate.limit tenant=t_195 remaining=17 window=60s +2026-07-01T08:03:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000196 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:17Z DEBUG cache.miss key=report:197 shard=3 ttl=900 +2026-07-01T08:03:18Z INFO worker.job id=198 queue=exports attempt=1 result=ok +2026-07-01T08:03:19Z WARN rate.limit tenant=t_199 remaining=17 window=60s +2026-07-01T08:03:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000200 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:21Z DEBUG cache.miss key=report:201 shard=3 ttl=900 +2026-07-01T08:03:22Z INFO worker.job id=202 queue=exports attempt=1 result=ok +2026-07-01T08:03:23Z WARN rate.limit tenant=t_203 remaining=17 window=60s +2026-07-01T08:03:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000204 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:25Z DEBUG cache.miss key=report:205 shard=3 ttl=900 +2026-07-01T08:03:26Z INFO worker.job id=206 queue=exports attempt=1 result=ok +2026-07-01T08:03:27Z WARN rate.limit tenant=t_207 remaining=17 window=60s +2026-07-01T08:03:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000208 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:29Z DEBUG cache.miss key=report:209 shard=3 ttl=900 +2026-07-01T08:03:30Z INFO worker.job id=210 queue=exports attempt=1 result=ok +2026-07-01T08:03:31Z WARN rate.limit tenant=t_211 remaining=17 window=60s +2026-07-01T08:03:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000212 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:33Z DEBUG cache.miss key=report:213 shard=3 ttl=900 +2026-07-01T08:03:34Z INFO worker.job id=214 queue=exports attempt=1 result=ok +2026-07-01T08:03:35Z WARN rate.limit tenant=t_215 remaining=17 window=60s +2026-07-01T08:03:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000216 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:37Z DEBUG cache.miss key=report:217 shard=3 ttl=900 +2026-07-01T08:03:38Z INFO worker.job id=218 queue=exports attempt=1 result=ok +2026-07-01T08:03:39Z WARN rate.limit tenant=t_219 remaining=17 window=60s +2026-07-01T08:03:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000220 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:41Z DEBUG cache.miss key=report:221 shard=3 ttl=900 +2026-07-01T08:03:42Z INFO worker.job id=222 queue=exports attempt=1 result=ok +2026-07-01T08:03:43Z WARN rate.limit tenant=t_223 remaining=17 window=60s +2026-07-01T08:03:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000224 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:45Z DEBUG cache.miss key=report:225 shard=3 ttl=900 +2026-07-01T08:03:46Z INFO worker.job id=226 queue=exports attempt=1 result=ok +2026-07-01T08:03:47Z WARN rate.limit tenant=t_227 remaining=17 window=60s +2026-07-01T08:03:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000228 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:49Z DEBUG cache.miss key=report:229 shard=3 ttl=900 +2026-07-01T08:03:50Z INFO worker.job id=230 queue=exports attempt=1 result=ok +2026-07-01T08:03:51Z WARN rate.limit tenant=t_231 remaining=17 window=60s +2026-07-01T08:03:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000232 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:53Z DEBUG cache.miss key=report:233 shard=3 ttl=900 +2026-07-01T08:03:54Z INFO worker.job id=234 queue=exports attempt=1 result=ok +2026-07-01T08:03:55Z WARN rate.limit tenant=t_235 remaining=17 window=60s +2026-07-01T08:03:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000236 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:03:57Z DEBUG cache.miss key=report:237 shard=3 ttl=900 +2026-07-01T08:03:58Z INFO worker.job id=238 queue=exports attempt=1 result=ok +2026-07-01T08:03:59Z WARN rate.limit tenant=t_239 remaining=17 window=60s +2026-07-01T08:04:00Z WARN pii.echo user_email=billing-team@example.invalid callback=+1 646 555 0123 +2026-07-01T08:04:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000240 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:01Z DEBUG cache.miss key=report:241 shard=3 ttl=900 +2026-07-01T08:04:02Z INFO worker.job id=242 queue=exports attempt=1 result=ok +2026-07-01T08:04:03Z WARN rate.limit tenant=t_243 remaining=17 window=60s +2026-07-01T08:04:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000244 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:05Z DEBUG cache.miss key=report:245 shard=3 ttl=900 +2026-07-01T08:04:06Z INFO worker.job id=246 queue=exports attempt=1 result=ok +2026-07-01T08:04:07Z WARN rate.limit tenant=t_247 remaining=17 window=60s +2026-07-01T08:04:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000248 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:09Z DEBUG cache.miss key=report:249 shard=3 ttl=900 +2026-07-01T08:04:10Z INFO worker.job id=250 queue=exports attempt=1 result=ok +2026-07-01T08:04:11Z WARN rate.limit tenant=t_251 remaining=17 window=60s +2026-07-01T08:04:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000252 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:13Z DEBUG cache.miss key=report:253 shard=3 ttl=900 +2026-07-01T08:04:14Z INFO worker.job id=254 queue=exports attempt=1 result=ok +2026-07-01T08:04:15Z WARN rate.limit tenant=t_255 remaining=17 window=60s +2026-07-01T08:04:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000256 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:17Z DEBUG cache.miss key=report:257 shard=3 ttl=900 +2026-07-01T08:04:18Z INFO worker.job id=258 queue=exports attempt=1 result=ok +2026-07-01T08:04:19Z WARN rate.limit tenant=t_259 remaining=17 window=60s +2026-07-01T08:04:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000260 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:21Z DEBUG cache.miss key=report:261 shard=3 ttl=900 +2026-07-01T08:04:22Z INFO worker.job id=262 queue=exports attempt=1 result=ok +2026-07-01T08:04:23Z WARN rate.limit tenant=t_263 remaining=17 window=60s +2026-07-01T08:04:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000264 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:25Z DEBUG cache.miss key=report:265 shard=3 ttl=900 +2026-07-01T08:04:26Z INFO worker.job id=266 queue=exports attempt=1 result=ok +2026-07-01T08:04:27Z WARN rate.limit tenant=t_267 remaining=17 window=60s +2026-07-01T08:04:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000268 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:29Z DEBUG cache.miss key=report:269 shard=3 ttl=900 +2026-07-01T08:04:30Z INFO worker.job id=270 queue=exports attempt=1 result=ok +2026-07-01T08:04:31Z WARN rate.limit tenant=t_271 remaining=17 window=60s +2026-07-01T08:04:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000272 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:33Z DEBUG cache.miss key=report:273 shard=3 ttl=900 +2026-07-01T08:04:34Z INFO worker.job id=274 queue=exports attempt=1 result=ok +2026-07-01T08:04:35Z WARN rate.limit tenant=t_275 remaining=17 window=60s +2026-07-01T08:04:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000276 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:37Z DEBUG cache.miss key=report:277 shard=3 ttl=900 +2026-07-01T08:04:38Z INFO worker.job id=278 queue=exports attempt=1 result=ok +2026-07-01T08:04:39Z WARN rate.limit tenant=t_279 remaining=17 window=60s +2026-07-01T08:04:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000280 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:41Z DEBUG cache.miss key=report:281 shard=3 ttl=900 +2026-07-01T08:04:42Z INFO worker.job id=282 queue=exports attempt=1 result=ok +2026-07-01T08:04:43Z WARN rate.limit tenant=t_283 remaining=17 window=60s +2026-07-01T08:04:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000284 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:45Z DEBUG cache.miss key=report:285 shard=3 ttl=900 +2026-07-01T08:04:46Z INFO worker.job id=286 queue=exports attempt=1 result=ok +2026-07-01T08:04:47Z WARN rate.limit tenant=t_287 remaining=17 window=60s +2026-07-01T08:04:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000288 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:49Z DEBUG cache.miss key=report:289 shard=3 ttl=900 +2026-07-01T08:04:50Z INFO worker.job id=290 queue=exports attempt=1 result=ok +2026-07-01T08:04:51Z WARN rate.limit tenant=t_291 remaining=17 window=60s +2026-07-01T08:04:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000292 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:53Z DEBUG cache.miss key=report:293 shard=3 ttl=900 +2026-07-01T08:04:54Z INFO worker.job id=294 queue=exports attempt=1 result=ok +2026-07-01T08:04:55Z WARN rate.limit tenant=t_295 remaining=17 window=60s +2026-07-01T08:04:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000296 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:04:57Z DEBUG cache.miss key=report:297 shard=3 ttl=900 +2026-07-01T08:04:58Z INFO worker.job id=298 queue=exports attempt=1 result=ok +2026-07-01T08:04:59Z WARN rate.limit tenant=t_299 remaining=17 window=60s +2026-07-01T08:05:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000300 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:01Z DEBUG cache.miss key=report:301 shard=3 ttl=900 +2026-07-01T08:05:02Z INFO worker.job id=302 queue=exports attempt=1 result=ok +2026-07-01T08:05:03Z WARN rate.limit tenant=t_303 remaining=17 window=60s +2026-07-01T08:05:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000304 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:05Z DEBUG cache.miss key=report:305 shard=3 ttl=900 +2026-07-01T08:05:06Z INFO worker.job id=306 queue=exports attempt=1 result=ok +2026-07-01T08:05:07Z WARN rate.limit tenant=t_307 remaining=17 window=60s +2026-07-01T08:05:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000308 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:09Z DEBUG cache.miss key=report:309 shard=3 ttl=900 +2026-07-01T08:05:10Z INFO worker.job id=310 queue=exports attempt=1 result=ok +2026-07-01T08:05:11Z WARN rate.limit tenant=t_311 remaining=17 window=60s +2026-07-01T08:05:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000312 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:13Z DEBUG cache.miss key=report:313 shard=3 ttl=900 +2026-07-01T08:05:14Z INFO worker.job id=314 queue=exports attempt=1 result=ok +2026-07-01T08:05:15Z WARN rate.limit tenant=t_315 remaining=17 window=60s +2026-07-01T08:05:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000316 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:17Z DEBUG cache.miss key=report:317 shard=3 ttl=900 +2026-07-01T08:05:18Z INFO worker.job id=318 queue=exports attempt=1 result=ok +2026-07-01T08:05:19Z WARN rate.limit tenant=t_319 remaining=17 window=60s +2026-07-01T08:05:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000320 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:21Z DEBUG cache.miss key=report:321 shard=3 ttl=900 +2026-07-01T08:05:22Z INFO worker.job id=322 queue=exports attempt=1 result=ok +2026-07-01T08:05:23Z WARN rate.limit tenant=t_323 remaining=17 window=60s +2026-07-01T08:05:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000324 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:25Z DEBUG cache.miss key=report:325 shard=3 ttl=900 +2026-07-01T08:05:26Z INFO worker.job id=326 queue=exports attempt=1 result=ok +2026-07-01T08:05:27Z WARN rate.limit tenant=t_327 remaining=17 window=60s +2026-07-01T08:05:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000328 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:29Z DEBUG cache.miss key=report:329 shard=3 ttl=900 +2026-07-01T08:05:30Z INFO worker.job id=330 queue=exports attempt=1 result=ok +2026-07-01T08:05:31Z WARN rate.limit tenant=t_331 remaining=17 window=60s +2026-07-01T08:05:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000332 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:33Z DEBUG cache.miss key=report:333 shard=3 ttl=900 +2026-07-01T08:05:34Z INFO worker.job id=334 queue=exports attempt=1 result=ok +2026-07-01T08:05:35Z WARN rate.limit tenant=t_335 remaining=17 window=60s +2026-07-01T08:05:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000336 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:37Z DEBUG cache.miss key=report:337 shard=3 ttl=900 +2026-07-01T08:05:38Z INFO worker.job id=338 queue=exports attempt=1 result=ok +2026-07-01T08:05:39Z WARN rate.limit tenant=t_339 remaining=17 window=60s +2026-07-01T08:05:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000340 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:41Z DEBUG cache.miss key=report:341 shard=3 ttl=900 +2026-07-01T08:05:42Z INFO worker.job id=342 queue=exports attempt=1 result=ok +2026-07-01T08:05:43Z WARN rate.limit tenant=t_343 remaining=17 window=60s +2026-07-01T08:05:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000344 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:45Z DEBUG cache.miss key=report:345 shard=3 ttl=900 +2026-07-01T08:05:46Z INFO worker.job id=346 queue=exports attempt=1 result=ok +2026-07-01T08:05:47Z WARN rate.limit tenant=t_347 remaining=17 window=60s +2026-07-01T08:05:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000348 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:49Z DEBUG cache.miss key=report:349 shard=3 ttl=900 +2026-07-01T08:05:50Z INFO worker.job id=350 queue=exports attempt=1 result=ok +2026-07-01T08:05:51Z WARN rate.limit tenant=t_351 remaining=17 window=60s +2026-07-01T08:05:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000352 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:53Z DEBUG cache.miss key=report:353 shard=3 ttl=900 +2026-07-01T08:05:54Z INFO worker.job id=354 queue=exports attempt=1 result=ok +2026-07-01T08:05:55Z WARN rate.limit tenant=t_355 remaining=17 window=60s +2026-07-01T08:05:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000356 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:05:57Z DEBUG cache.miss key=report:357 shard=3 ttl=900 +2026-07-01T08:05:58Z INFO worker.job id=358 queue=exports attempt=1 result=ok +2026-07-01T08:05:59Z WARN rate.limit tenant=t_359 remaining=17 window=60s +2026-07-01T08:06:00Z WARN pii.echo user_email=maria.gonzalez@example.com callback=(303) 555-0176 +2026-07-01T08:06:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000360 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:01Z DEBUG cache.miss key=report:361 shard=3 ttl=900 +2026-07-01T08:06:02Z INFO worker.job id=362 queue=exports attempt=1 result=ok +2026-07-01T08:06:03Z WARN rate.limit tenant=t_363 remaining=17 window=60s +2026-07-01T08:06:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000364 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:05Z DEBUG cache.miss key=report:365 shard=3 ttl=900 +2026-07-01T08:06:06Z INFO worker.job id=366 queue=exports attempt=1 result=ok +2026-07-01T08:06:07Z WARN rate.limit tenant=t_367 remaining=17 window=60s +2026-07-01T08:06:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000368 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:09Z DEBUG cache.miss key=report:369 shard=3 ttl=900 +2026-07-01T08:06:10Z INFO worker.job id=370 queue=exports attempt=1 result=ok +2026-07-01T08:06:11Z WARN rate.limit tenant=t_371 remaining=17 window=60s +2026-07-01T08:06:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000372 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:13Z DEBUG cache.miss key=report:373 shard=3 ttl=900 +2026-07-01T08:06:14Z INFO worker.job id=374 queue=exports attempt=1 result=ok +2026-07-01T08:06:15Z WARN rate.limit tenant=t_375 remaining=17 window=60s +2026-07-01T08:06:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000376 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:17Z DEBUG cache.miss key=report:377 shard=3 ttl=900 +2026-07-01T08:06:18Z INFO worker.job id=378 queue=exports attempt=1 result=ok +2026-07-01T08:06:19Z WARN rate.limit tenant=t_379 remaining=17 window=60s +2026-07-01T08:06:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000380 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:21Z DEBUG cache.miss key=report:381 shard=3 ttl=900 +2026-07-01T08:06:22Z INFO worker.job id=382 queue=exports attempt=1 result=ok +2026-07-01T08:06:23Z WARN rate.limit tenant=t_383 remaining=17 window=60s +2026-07-01T08:06:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000384 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:25Z DEBUG cache.miss key=report:385 shard=3 ttl=900 +2026-07-01T08:06:26Z INFO worker.job id=386 queue=exports attempt=1 result=ok +2026-07-01T08:06:27Z WARN rate.limit tenant=t_387 remaining=17 window=60s +2026-07-01T08:06:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000388 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:29Z DEBUG cache.miss key=report:389 shard=3 ttl=900 +2026-07-01T08:06:30Z INFO worker.job id=390 queue=exports attempt=1 result=ok +2026-07-01T08:06:31Z WARN rate.limit tenant=t_391 remaining=17 window=60s +2026-07-01T08:06:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000392 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:33Z DEBUG cache.miss key=report:393 shard=3 ttl=900 +2026-07-01T08:06:34Z INFO worker.job id=394 queue=exports attempt=1 result=ok +2026-07-01T08:06:35Z WARN rate.limit tenant=t_395 remaining=17 window=60s +2026-07-01T08:06:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000396 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:37Z DEBUG cache.miss key=report:397 shard=3 ttl=900 +2026-07-01T08:06:38Z INFO worker.job id=398 queue=exports attempt=1 result=ok +2026-07-01T08:06:39Z WARN rate.limit tenant=t_399 remaining=17 window=60s +2026-07-01T08:06:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000400 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:41Z DEBUG cache.miss key=report:401 shard=3 ttl=900 +2026-07-01T08:06:42Z INFO worker.job id=402 queue=exports attempt=1 result=ok +2026-07-01T08:06:43Z WARN rate.limit tenant=t_403 remaining=17 window=60s +2026-07-01T08:06:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000404 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:45Z DEBUG cache.miss key=report:405 shard=3 ttl=900 +2026-07-01T08:06:46Z INFO worker.job id=406 queue=exports attempt=1 result=ok +2026-07-01T08:06:47Z WARN rate.limit tenant=t_407 remaining=17 window=60s +2026-07-01T08:06:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000408 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:49Z DEBUG cache.miss key=report:409 shard=3 ttl=900 +2026-07-01T08:06:50Z INFO worker.job id=410 queue=exports attempt=1 result=ok +2026-07-01T08:06:51Z WARN rate.limit tenant=t_411 remaining=17 window=60s +2026-07-01T08:06:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000412 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:53Z DEBUG cache.miss key=report:413 shard=3 ttl=900 +2026-07-01T08:06:54Z INFO worker.job id=414 queue=exports attempt=1 result=ok +2026-07-01T08:06:55Z WARN rate.limit tenant=t_415 remaining=17 window=60s +2026-07-01T08:06:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000416 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:06:57Z DEBUG cache.miss key=report:417 shard=3 ttl=900 +2026-07-01T08:06:58Z INFO worker.job id=418 queue=exports attempt=1 result=ok +2026-07-01T08:06:59Z WARN rate.limit tenant=t_419 remaining=17 window=60s +2026-07-01T08:07:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000420 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:01Z DEBUG cache.miss key=report:421 shard=3 ttl=900 +2026-07-01T08:07:02Z INFO worker.job id=422 queue=exports attempt=1 result=ok +2026-07-01T08:07:03Z WARN rate.limit tenant=t_423 remaining=17 window=60s +2026-07-01T08:07:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000424 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:05Z DEBUG cache.miss key=report:425 shard=3 ttl=900 +2026-07-01T08:07:06Z INFO worker.job id=426 queue=exports attempt=1 result=ok +2026-07-01T08:07:07Z WARN rate.limit tenant=t_427 remaining=17 window=60s +2026-07-01T08:07:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000428 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:09Z DEBUG cache.miss key=report:429 shard=3 ttl=900 +2026-07-01T08:07:10Z INFO worker.job id=430 queue=exports attempt=1 result=ok +2026-07-01T08:07:11Z WARN rate.limit tenant=t_431 remaining=17 window=60s +2026-07-01T08:07:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000432 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:13Z DEBUG cache.miss key=report:433 shard=3 ttl=900 +2026-07-01T08:07:14Z INFO worker.job id=434 queue=exports attempt=1 result=ok +2026-07-01T08:07:15Z WARN rate.limit tenant=t_435 remaining=17 window=60s +2026-07-01T08:07:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000436 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:17Z DEBUG cache.miss key=report:437 shard=3 ttl=900 +2026-07-01T08:07:18Z INFO worker.job id=438 queue=exports attempt=1 result=ok +2026-07-01T08:07:19Z WARN rate.limit tenant=t_439 remaining=17 window=60s +2026-07-01T08:07:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000440 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:21Z DEBUG cache.miss key=report:441 shard=3 ttl=900 +2026-07-01T08:07:22Z INFO worker.job id=442 queue=exports attempt=1 result=ok +2026-07-01T08:07:23Z WARN rate.limit tenant=t_443 remaining=17 window=60s +2026-07-01T08:07:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000444 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:25Z DEBUG cache.miss key=report:445 shard=3 ttl=900 +2026-07-01T08:07:26Z INFO worker.job id=446 queue=exports attempt=1 result=ok +2026-07-01T08:07:27Z WARN rate.limit tenant=t_447 remaining=17 window=60s +2026-07-01T08:07:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000448 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:29Z DEBUG cache.miss key=report:449 shard=3 ttl=900 +2026-07-01T08:07:30Z INFO worker.job id=450 queue=exports attempt=1 result=ok +2026-07-01T08:07:31Z WARN rate.limit tenant=t_451 remaining=17 window=60s +2026-07-01T08:07:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000452 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:33Z DEBUG cache.miss key=report:453 shard=3 ttl=900 +2026-07-01T08:07:34Z INFO worker.job id=454 queue=exports attempt=1 result=ok +2026-07-01T08:07:35Z WARN rate.limit tenant=t_455 remaining=17 window=60s +2026-07-01T08:07:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000456 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:37Z DEBUG cache.miss key=report:457 shard=3 ttl=900 +2026-07-01T08:07:38Z INFO worker.job id=458 queue=exports attempt=1 result=ok +2026-07-01T08:07:39Z WARN rate.limit tenant=t_459 remaining=17 window=60s +2026-07-01T08:07:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000460 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:41Z DEBUG cache.miss key=report:461 shard=3 ttl=900 +2026-07-01T08:07:42Z INFO worker.job id=462 queue=exports attempt=1 result=ok +2026-07-01T08:07:43Z WARN rate.limit tenant=t_463 remaining=17 window=60s +2026-07-01T08:07:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000464 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:45Z DEBUG cache.miss key=report:465 shard=3 ttl=900 +2026-07-01T08:07:46Z INFO worker.job id=466 queue=exports attempt=1 result=ok +2026-07-01T08:07:47Z WARN rate.limit tenant=t_467 remaining=17 window=60s +2026-07-01T08:07:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000468 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:49Z DEBUG cache.miss key=report:469 shard=3 ttl=900 +2026-07-01T08:07:50Z INFO worker.job id=470 queue=exports attempt=1 result=ok +2026-07-01T08:07:51Z WARN rate.limit tenant=t_471 remaining=17 window=60s +2026-07-01T08:07:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000472 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:53Z DEBUG cache.miss key=report:473 shard=3 ttl=900 +2026-07-01T08:07:54Z INFO worker.job id=474 queue=exports attempt=1 result=ok +2026-07-01T08:07:55Z WARN rate.limit tenant=t_475 remaining=17 window=60s +2026-07-01T08:07:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000476 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:07:57Z DEBUG cache.miss key=report:477 shard=3 ttl=900 +2026-07-01T08:07:58Z INFO worker.job id=478 queue=exports attempt=1 result=ok +2026-07-01T08:07:59Z WARN rate.limit tenant=t_479 remaining=17 window=60s +2026-07-01T08:08:00Z WARN pii.echo user_email=alex.rivera@example.invalid callback=(212) 555-0142 +2026-07-01T08:08:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000480 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:01Z DEBUG cache.miss key=report:481 shard=3 ttl=900 +2026-07-01T08:08:02Z INFO worker.job id=482 queue=exports attempt=1 result=ok +2026-07-01T08:08:03Z WARN rate.limit tenant=t_483 remaining=17 window=60s +2026-07-01T08:08:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000484 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:05Z DEBUG cache.miss key=report:485 shard=3 ttl=900 +2026-07-01T08:08:06Z INFO worker.job id=486 queue=exports attempt=1 result=ok +2026-07-01T08:08:07Z WARN rate.limit tenant=t_487 remaining=17 window=60s +2026-07-01T08:08:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000488 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:09Z DEBUG cache.miss key=report:489 shard=3 ttl=900 +2026-07-01T08:08:10Z INFO worker.job id=490 queue=exports attempt=1 result=ok +2026-07-01T08:08:11Z WARN rate.limit tenant=t_491 remaining=17 window=60s +2026-07-01T08:08:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000492 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:13Z DEBUG cache.miss key=report:493 shard=3 ttl=900 +2026-07-01T08:08:14Z INFO worker.job id=494 queue=exports attempt=1 result=ok +2026-07-01T08:08:15Z WARN rate.limit tenant=t_495 remaining=17 window=60s +2026-07-01T08:08:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000496 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:17Z DEBUG cache.miss key=report:497 shard=3 ttl=900 +2026-07-01T08:08:18Z INFO worker.job id=498 queue=exports attempt=1 result=ok +2026-07-01T08:08:19Z WARN rate.limit tenant=t_499 remaining=17 window=60s +2026-07-01T08:08:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000500 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:21Z DEBUG cache.miss key=report:501 shard=3 ttl=900 +2026-07-01T08:08:22Z INFO worker.job id=502 queue=exports attempt=1 result=ok +2026-07-01T08:08:23Z WARN rate.limit tenant=t_503 remaining=17 window=60s +2026-07-01T08:08:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000504 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:25Z DEBUG cache.miss key=report:505 shard=3 ttl=900 +2026-07-01T08:08:26Z INFO worker.job id=506 queue=exports attempt=1 result=ok +2026-07-01T08:08:27Z WARN rate.limit tenant=t_507 remaining=17 window=60s +2026-07-01T08:08:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000508 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:29Z DEBUG cache.miss key=report:509 shard=3 ttl=900 +2026-07-01T08:08:30Z INFO worker.job id=510 queue=exports attempt=1 result=ok +2026-07-01T08:08:31Z WARN rate.limit tenant=t_511 remaining=17 window=60s +2026-07-01T08:08:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000512 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:33Z DEBUG cache.miss key=report:513 shard=3 ttl=900 +2026-07-01T08:08:34Z INFO worker.job id=514 queue=exports attempt=1 result=ok +2026-07-01T08:08:35Z WARN rate.limit tenant=t_515 remaining=17 window=60s +2026-07-01T08:08:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000516 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:37Z DEBUG cache.miss key=report:517 shard=3 ttl=900 +2026-07-01T08:08:38Z INFO worker.job id=518 queue=exports attempt=1 result=ok +2026-07-01T08:08:39Z WARN rate.limit tenant=t_519 remaining=17 window=60s +2026-07-01T08:08:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000520 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:41Z DEBUG cache.miss key=report:521 shard=3 ttl=900 +2026-07-01T08:08:42Z INFO worker.job id=522 queue=exports attempt=1 result=ok +2026-07-01T08:08:43Z WARN rate.limit tenant=t_523 remaining=17 window=60s +2026-07-01T08:08:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000524 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:45Z DEBUG cache.miss key=report:525 shard=3 ttl=900 +2026-07-01T08:08:46Z INFO worker.job id=526 queue=exports attempt=1 result=ok +2026-07-01T08:08:47Z WARN rate.limit tenant=t_527 remaining=17 window=60s +2026-07-01T08:08:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000528 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:49Z DEBUG cache.miss key=report:529 shard=3 ttl=900 +2026-07-01T08:08:50Z INFO worker.job id=530 queue=exports attempt=1 result=ok +2026-07-01T08:08:51Z WARN rate.limit tenant=t_531 remaining=17 window=60s +2026-07-01T08:08:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000532 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:53Z DEBUG cache.miss key=report:533 shard=3 ttl=900 +2026-07-01T08:08:54Z INFO worker.job id=534 queue=exports attempt=1 result=ok +2026-07-01T08:08:55Z WARN rate.limit tenant=t_535 remaining=17 window=60s +2026-07-01T08:08:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000536 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:08:57Z DEBUG cache.miss key=report:537 shard=3 ttl=900 +2026-07-01T08:08:58Z INFO worker.job id=538 queue=exports attempt=1 result=ok +2026-07-01T08:08:59Z WARN rate.limit tenant=t_539 remaining=17 window=60s +2026-07-01T08:09:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000540 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:01Z DEBUG cache.miss key=report:541 shard=3 ttl=900 +2026-07-01T08:09:02Z INFO worker.job id=542 queue=exports attempt=1 result=ok +2026-07-01T08:09:03Z WARN rate.limit tenant=t_543 remaining=17 window=60s +2026-07-01T08:09:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000544 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:05Z DEBUG cache.miss key=report:545 shard=3 ttl=900 +2026-07-01T08:09:06Z INFO worker.job id=546 queue=exports attempt=1 result=ok +2026-07-01T08:09:07Z WARN rate.limit tenant=t_547 remaining=17 window=60s +2026-07-01T08:09:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000548 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:09Z DEBUG cache.miss key=report:549 shard=3 ttl=900 +2026-07-01T08:09:10Z INFO worker.job id=550 queue=exports attempt=1 result=ok +2026-07-01T08:09:11Z WARN rate.limit tenant=t_551 remaining=17 window=60s +2026-07-01T08:09:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000552 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:13Z DEBUG cache.miss key=report:553 shard=3 ttl=900 +2026-07-01T08:09:14Z INFO worker.job id=554 queue=exports attempt=1 result=ok +2026-07-01T08:09:15Z WARN rate.limit tenant=t_555 remaining=17 window=60s +2026-07-01T08:09:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000556 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:17Z DEBUG cache.miss key=report:557 shard=3 ttl=900 +2026-07-01T08:09:18Z INFO worker.job id=558 queue=exports attempt=1 result=ok +2026-07-01T08:09:19Z WARN rate.limit tenant=t_559 remaining=17 window=60s +2026-07-01T08:09:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000560 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:21Z DEBUG cache.miss key=report:561 shard=3 ttl=900 +2026-07-01T08:09:22Z INFO worker.job id=562 queue=exports attempt=1 result=ok +2026-07-01T08:09:23Z WARN rate.limit tenant=t_563 remaining=17 window=60s +2026-07-01T08:09:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000564 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:25Z DEBUG cache.miss key=report:565 shard=3 ttl=900 +2026-07-01T08:09:26Z INFO worker.job id=566 queue=exports attempt=1 result=ok +2026-07-01T08:09:27Z WARN rate.limit tenant=t_567 remaining=17 window=60s +2026-07-01T08:09:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000568 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:29Z DEBUG cache.miss key=report:569 shard=3 ttl=900 +2026-07-01T08:09:30Z INFO worker.job id=570 queue=exports attempt=1 result=ok +2026-07-01T08:09:31Z WARN rate.limit tenant=t_571 remaining=17 window=60s +2026-07-01T08:09:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000572 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:33Z DEBUG cache.miss key=report:573 shard=3 ttl=900 +2026-07-01T08:09:34Z INFO worker.job id=574 queue=exports attempt=1 result=ok +2026-07-01T08:09:35Z WARN rate.limit tenant=t_575 remaining=17 window=60s +2026-07-01T08:09:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000576 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:37Z DEBUG cache.miss key=report:577 shard=3 ttl=900 +2026-07-01T08:09:38Z INFO worker.job id=578 queue=exports attempt=1 result=ok +2026-07-01T08:09:39Z WARN rate.limit tenant=t_579 remaining=17 window=60s +2026-07-01T08:09:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000580 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:41Z DEBUG cache.miss key=report:581 shard=3 ttl=900 +2026-07-01T08:09:42Z INFO worker.job id=582 queue=exports attempt=1 result=ok +2026-07-01T08:09:43Z WARN rate.limit tenant=t_583 remaining=17 window=60s +2026-07-01T08:09:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000584 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:45Z DEBUG cache.miss key=report:585 shard=3 ttl=900 +2026-07-01T08:09:46Z INFO worker.job id=586 queue=exports attempt=1 result=ok +2026-07-01T08:09:47Z WARN rate.limit tenant=t_587 remaining=17 window=60s +2026-07-01T08:09:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000588 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:49Z DEBUG cache.miss key=report:589 shard=3 ttl=900 +2026-07-01T08:09:50Z INFO worker.job id=590 queue=exports attempt=1 result=ok +2026-07-01T08:09:51Z WARN rate.limit tenant=t_591 remaining=17 window=60s +2026-07-01T08:09:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000592 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:53Z DEBUG cache.miss key=report:593 shard=3 ttl=900 +2026-07-01T08:09:54Z INFO worker.job id=594 queue=exports attempt=1 result=ok +2026-07-01T08:09:55Z WARN rate.limit tenant=t_595 remaining=17 window=60s +2026-07-01T08:09:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000596 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:09:57Z DEBUG cache.miss key=report:597 shard=3 ttl=900 +2026-07-01T08:09:58Z INFO worker.job id=598 queue=exports attempt=1 result=ok +2026-07-01T08:09:59Z WARN rate.limit tenant=t_599 remaining=17 window=60s +2026-07-01T08:10:00Z WARN pii.echo user_email=j.chen@support.example.com callback=415-555-0188 +2026-07-01T08:10:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000600 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:01Z DEBUG cache.miss key=report:601 shard=3 ttl=900 +2026-07-01T08:10:02Z INFO worker.job id=602 queue=exports attempt=1 result=ok +2026-07-01T08:10:03Z WARN rate.limit tenant=t_603 remaining=17 window=60s +2026-07-01T08:10:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000604 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:05Z DEBUG cache.miss key=report:605 shard=3 ttl=900 +2026-07-01T08:10:06Z INFO worker.job id=606 queue=exports attempt=1 result=ok +2026-07-01T08:10:07Z WARN rate.limit tenant=t_607 remaining=17 window=60s +2026-07-01T08:10:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000608 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:09Z DEBUG cache.miss key=report:609 shard=3 ttl=900 +2026-07-01T08:10:10Z INFO worker.job id=610 queue=exports attempt=1 result=ok +2026-07-01T08:10:11Z WARN rate.limit tenant=t_611 remaining=17 window=60s +2026-07-01T08:10:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000612 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:13Z DEBUG cache.miss key=report:613 shard=3 ttl=900 +2026-07-01T08:10:14Z INFO worker.job id=614 queue=exports attempt=1 result=ok +2026-07-01T08:10:15Z WARN rate.limit tenant=t_615 remaining=17 window=60s +2026-07-01T08:10:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000616 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:17Z DEBUG cache.miss key=report:617 shard=3 ttl=900 +2026-07-01T08:10:18Z INFO worker.job id=618 queue=exports attempt=1 result=ok +2026-07-01T08:10:19Z WARN rate.limit tenant=t_619 remaining=17 window=60s +2026-07-01T08:10:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000620 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:21Z DEBUG cache.miss key=report:621 shard=3 ttl=900 +2026-07-01T08:10:22Z INFO worker.job id=622 queue=exports attempt=1 result=ok +2026-07-01T08:10:23Z WARN rate.limit tenant=t_623 remaining=17 window=60s +2026-07-01T08:10:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000624 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:25Z DEBUG cache.miss key=report:625 shard=3 ttl=900 +2026-07-01T08:10:26Z INFO worker.job id=626 queue=exports attempt=1 result=ok +2026-07-01T08:10:27Z WARN rate.limit tenant=t_627 remaining=17 window=60s +2026-07-01T08:10:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000628 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:29Z DEBUG cache.miss key=report:629 shard=3 ttl=900 +2026-07-01T08:10:30Z INFO worker.job id=630 queue=exports attempt=1 result=ok +2026-07-01T08:10:31Z WARN rate.limit tenant=t_631 remaining=17 window=60s +2026-07-01T08:10:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000632 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:33Z DEBUG cache.miss key=report:633 shard=3 ttl=900 +2026-07-01T08:10:34Z INFO worker.job id=634 queue=exports attempt=1 result=ok +2026-07-01T08:10:35Z WARN rate.limit tenant=t_635 remaining=17 window=60s +2026-07-01T08:10:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000636 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:37Z DEBUG cache.miss key=report:637 shard=3 ttl=900 +2026-07-01T08:10:38Z INFO worker.job id=638 queue=exports attempt=1 result=ok +2026-07-01T08:10:39Z WARN rate.limit tenant=t_639 remaining=17 window=60s +2026-07-01T08:10:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000640 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:41Z DEBUG cache.miss key=report:641 shard=3 ttl=900 +2026-07-01T08:10:42Z INFO worker.job id=642 queue=exports attempt=1 result=ok +2026-07-01T08:10:43Z WARN rate.limit tenant=t_643 remaining=17 window=60s +2026-07-01T08:10:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000644 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:45Z DEBUG cache.miss key=report:645 shard=3 ttl=900 +2026-07-01T08:10:46Z INFO worker.job id=646 queue=exports attempt=1 result=ok +2026-07-01T08:10:47Z WARN rate.limit tenant=t_647 remaining=17 window=60s +2026-07-01T08:10:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000648 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:49Z DEBUG cache.miss key=report:649 shard=3 ttl=900 +2026-07-01T08:10:50Z INFO worker.job id=650 queue=exports attempt=1 result=ok +2026-07-01T08:10:51Z WARN rate.limit tenant=t_651 remaining=17 window=60s +2026-07-01T08:10:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000652 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:53Z DEBUG cache.miss key=report:653 shard=3 ttl=900 +2026-07-01T08:10:54Z INFO worker.job id=654 queue=exports attempt=1 result=ok +2026-07-01T08:10:55Z WARN rate.limit tenant=t_655 remaining=17 window=60s +2026-07-01T08:10:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000656 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:10:57Z DEBUG cache.miss key=report:657 shard=3 ttl=900 +2026-07-01T08:10:58Z INFO worker.job id=658 queue=exports attempt=1 result=ok +2026-07-01T08:10:59Z WARN rate.limit tenant=t_659 remaining=17 window=60s +2026-07-01T08:11:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000660 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:01Z DEBUG cache.miss key=report:661 shard=3 ttl=900 +2026-07-01T08:11:02Z INFO worker.job id=662 queue=exports attempt=1 result=ok +2026-07-01T08:11:03Z WARN rate.limit tenant=t_663 remaining=17 window=60s +2026-07-01T08:11:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000664 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:05Z DEBUG cache.miss key=report:665 shard=3 ttl=900 +2026-07-01T08:11:06Z INFO worker.job id=666 queue=exports attempt=1 result=ok +2026-07-01T08:11:07Z WARN rate.limit tenant=t_667 remaining=17 window=60s +2026-07-01T08:11:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000668 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:09Z DEBUG cache.miss key=report:669 shard=3 ttl=900 +2026-07-01T08:11:10Z INFO worker.job id=670 queue=exports attempt=1 result=ok +2026-07-01T08:11:11Z WARN rate.limit tenant=t_671 remaining=17 window=60s +2026-07-01T08:11:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000672 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:13Z DEBUG cache.miss key=report:673 shard=3 ttl=900 +2026-07-01T08:11:14Z INFO worker.job id=674 queue=exports attempt=1 result=ok +2026-07-01T08:11:15Z WARN rate.limit tenant=t_675 remaining=17 window=60s +2026-07-01T08:11:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000676 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:17Z DEBUG cache.miss key=report:677 shard=3 ttl=900 +2026-07-01T08:11:18Z INFO worker.job id=678 queue=exports attempt=1 result=ok +2026-07-01T08:11:19Z WARN rate.limit tenant=t_679 remaining=17 window=60s +2026-07-01T08:11:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000680 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:21Z DEBUG cache.miss key=report:681 shard=3 ttl=900 +2026-07-01T08:11:22Z INFO worker.job id=682 queue=exports attempt=1 result=ok +2026-07-01T08:11:23Z WARN rate.limit tenant=t_683 remaining=17 window=60s +2026-07-01T08:11:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000684 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:25Z DEBUG cache.miss key=report:685 shard=3 ttl=900 +2026-07-01T08:11:26Z INFO worker.job id=686 queue=exports attempt=1 result=ok +2026-07-01T08:11:27Z WARN rate.limit tenant=t_687 remaining=17 window=60s +2026-07-01T08:11:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000688 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:29Z DEBUG cache.miss key=report:689 shard=3 ttl=900 +2026-07-01T08:11:30Z INFO worker.job id=690 queue=exports attempt=1 result=ok +2026-07-01T08:11:31Z WARN rate.limit tenant=t_691 remaining=17 window=60s +2026-07-01T08:11:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000692 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:33Z DEBUG cache.miss key=report:693 shard=3 ttl=900 +2026-07-01T08:11:34Z INFO worker.job id=694 queue=exports attempt=1 result=ok +2026-07-01T08:11:35Z WARN rate.limit tenant=t_695 remaining=17 window=60s +2026-07-01T08:11:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000696 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:37Z DEBUG cache.miss key=report:697 shard=3 ttl=900 +2026-07-01T08:11:38Z INFO worker.job id=698 queue=exports attempt=1 result=ok +2026-07-01T08:11:39Z WARN rate.limit tenant=t_699 remaining=17 window=60s +2026-07-01T08:11:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000700 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:41Z DEBUG cache.miss key=report:701 shard=3 ttl=900 +2026-07-01T08:11:42Z INFO worker.job id=702 queue=exports attempt=1 result=ok +2026-07-01T08:11:43Z WARN rate.limit tenant=t_703 remaining=17 window=60s +2026-07-01T08:11:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000704 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:45Z DEBUG cache.miss key=report:705 shard=3 ttl=900 +2026-07-01T08:11:46Z INFO worker.job id=706 queue=exports attempt=1 result=ok +2026-07-01T08:11:47Z WARN rate.limit tenant=t_707 remaining=17 window=60s +2026-07-01T08:11:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000708 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:49Z DEBUG cache.miss key=report:709 shard=3 ttl=900 +2026-07-01T08:11:50Z INFO worker.job id=710 queue=exports attempt=1 result=ok +2026-07-01T08:11:51Z WARN rate.limit tenant=t_711 remaining=17 window=60s +2026-07-01T08:11:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000712 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:53Z DEBUG cache.miss key=report:713 shard=3 ttl=900 +2026-07-01T08:11:54Z INFO worker.job id=714 queue=exports attempt=1 result=ok +2026-07-01T08:11:55Z WARN rate.limit tenant=t_715 remaining=17 window=60s +2026-07-01T08:11:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000716 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:11:57Z DEBUG cache.miss key=report:717 shard=3 ttl=900 +2026-07-01T08:11:58Z INFO worker.job id=718 queue=exports attempt=1 result=ok +2026-07-01T08:11:59Z WARN rate.limit tenant=t_719 remaining=17 window=60s +2026-07-01T08:12:00Z WARN pii.echo user_email=billing-team@example.invalid callback=+1 646 555 0123 +2026-07-01T08:12:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000720 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:01Z DEBUG cache.miss key=report:721 shard=3 ttl=900 +2026-07-01T08:12:02Z INFO worker.job id=722 queue=exports attempt=1 result=ok +2026-07-01T08:12:03Z WARN rate.limit tenant=t_723 remaining=17 window=60s +2026-07-01T08:12:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000724 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:05Z DEBUG cache.miss key=report:725 shard=3 ttl=900 +2026-07-01T08:12:06Z INFO worker.job id=726 queue=exports attempt=1 result=ok +2026-07-01T08:12:07Z WARN rate.limit tenant=t_727 remaining=17 window=60s +2026-07-01T08:12:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000728 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:09Z DEBUG cache.miss key=report:729 shard=3 ttl=900 +2026-07-01T08:12:10Z INFO worker.job id=730 queue=exports attempt=1 result=ok +2026-07-01T08:12:11Z WARN rate.limit tenant=t_731 remaining=17 window=60s +2026-07-01T08:12:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000732 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:13Z DEBUG cache.miss key=report:733 shard=3 ttl=900 +2026-07-01T08:12:14Z INFO worker.job id=734 queue=exports attempt=1 result=ok +2026-07-01T08:12:15Z WARN rate.limit tenant=t_735 remaining=17 window=60s +2026-07-01T08:12:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000736 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:17Z DEBUG cache.miss key=report:737 shard=3 ttl=900 +2026-07-01T08:12:18Z INFO worker.job id=738 queue=exports attempt=1 result=ok +2026-07-01T08:12:19Z WARN rate.limit tenant=t_739 remaining=17 window=60s +2026-07-01T08:12:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000740 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:21Z DEBUG cache.miss key=report:741 shard=3 ttl=900 +2026-07-01T08:12:22Z INFO worker.job id=742 queue=exports attempt=1 result=ok +2026-07-01T08:12:23Z WARN rate.limit tenant=t_743 remaining=17 window=60s +2026-07-01T08:12:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000744 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:25Z DEBUG cache.miss key=report:745 shard=3 ttl=900 +2026-07-01T08:12:26Z INFO worker.job id=746 queue=exports attempt=1 result=ok +2026-07-01T08:12:27Z WARN rate.limit tenant=t_747 remaining=17 window=60s +2026-07-01T08:12:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000748 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:29Z DEBUG cache.miss key=report:749 shard=3 ttl=900 +2026-07-01T08:12:30Z INFO worker.job id=750 queue=exports attempt=1 result=ok +2026-07-01T08:12:31Z WARN rate.limit tenant=t_751 remaining=17 window=60s +2026-07-01T08:12:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000752 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:33Z DEBUG cache.miss key=report:753 shard=3 ttl=900 +2026-07-01T08:12:34Z INFO worker.job id=754 queue=exports attempt=1 result=ok +2026-07-01T08:12:35Z WARN rate.limit tenant=t_755 remaining=17 window=60s +2026-07-01T08:12:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000756 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:37Z DEBUG cache.miss key=report:757 shard=3 ttl=900 +2026-07-01T08:12:38Z INFO worker.job id=758 queue=exports attempt=1 result=ok +2026-07-01T08:12:39Z WARN rate.limit tenant=t_759 remaining=17 window=60s +2026-07-01T08:12:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000760 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:41Z DEBUG cache.miss key=report:761 shard=3 ttl=900 +2026-07-01T08:12:42Z INFO worker.job id=762 queue=exports attempt=1 result=ok +2026-07-01T08:12:43Z WARN rate.limit tenant=t_763 remaining=17 window=60s +2026-07-01T08:12:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000764 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:45Z DEBUG cache.miss key=report:765 shard=3 ttl=900 +2026-07-01T08:12:46Z INFO worker.job id=766 queue=exports attempt=1 result=ok +2026-07-01T08:12:47Z WARN rate.limit tenant=t_767 remaining=17 window=60s +2026-07-01T08:12:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000768 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:49Z DEBUG cache.miss key=report:769 shard=3 ttl=900 +2026-07-01T08:12:50Z INFO worker.job id=770 queue=exports attempt=1 result=ok +2026-07-01T08:12:51Z WARN rate.limit tenant=t_771 remaining=17 window=60s +2026-07-01T08:12:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000772 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:53Z DEBUG cache.miss key=report:773 shard=3 ttl=900 +2026-07-01T08:12:54Z INFO worker.job id=774 queue=exports attempt=1 result=ok +2026-07-01T08:12:55Z WARN rate.limit tenant=t_775 remaining=17 window=60s +2026-07-01T08:12:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000776 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:12:57Z DEBUG cache.miss key=report:777 shard=3 ttl=900 +2026-07-01T08:12:58Z INFO worker.job id=778 queue=exports attempt=1 result=ok +2026-07-01T08:12:59Z WARN rate.limit tenant=t_779 remaining=17 window=60s +2026-07-01T08:13:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000780 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:01Z DEBUG cache.miss key=report:781 shard=3 ttl=900 +2026-07-01T08:13:02Z INFO worker.job id=782 queue=exports attempt=1 result=ok +2026-07-01T08:13:03Z WARN rate.limit tenant=t_783 remaining=17 window=60s +2026-07-01T08:13:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000784 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:05Z DEBUG cache.miss key=report:785 shard=3 ttl=900 +2026-07-01T08:13:06Z INFO worker.job id=786 queue=exports attempt=1 result=ok +2026-07-01T08:13:07Z WARN rate.limit tenant=t_787 remaining=17 window=60s +2026-07-01T08:13:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000788 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:09Z DEBUG cache.miss key=report:789 shard=3 ttl=900 +2026-07-01T08:13:10Z INFO worker.job id=790 queue=exports attempt=1 result=ok +2026-07-01T08:13:11Z WARN rate.limit tenant=t_791 remaining=17 window=60s +2026-07-01T08:13:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000792 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:13Z DEBUG cache.miss key=report:793 shard=3 ttl=900 +2026-07-01T08:13:14Z INFO worker.job id=794 queue=exports attempt=1 result=ok +2026-07-01T08:13:15Z WARN rate.limit tenant=t_795 remaining=17 window=60s +2026-07-01T08:13:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000796 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:17Z DEBUG cache.miss key=report:797 shard=3 ttl=900 +2026-07-01T08:13:18Z INFO worker.job id=798 queue=exports attempt=1 result=ok +2026-07-01T08:13:19Z WARN rate.limit tenant=t_799 remaining=17 window=60s +2026-07-01T08:13:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000800 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:21Z DEBUG cache.miss key=report:801 shard=3 ttl=900 +2026-07-01T08:13:22Z INFO worker.job id=802 queue=exports attempt=1 result=ok +2026-07-01T08:13:23Z WARN rate.limit tenant=t_803 remaining=17 window=60s +2026-07-01T08:13:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000804 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:25Z DEBUG cache.miss key=report:805 shard=3 ttl=900 +2026-07-01T08:13:26Z INFO worker.job id=806 queue=exports attempt=1 result=ok +2026-07-01T08:13:27Z WARN rate.limit tenant=t_807 remaining=17 window=60s +2026-07-01T08:13:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000808 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:29Z DEBUG cache.miss key=report:809 shard=3 ttl=900 +2026-07-01T08:13:30Z INFO worker.job id=810 queue=exports attempt=1 result=ok +2026-07-01T08:13:31Z WARN rate.limit tenant=t_811 remaining=17 window=60s +2026-07-01T08:13:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000812 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:33Z DEBUG cache.miss key=report:813 shard=3 ttl=900 +2026-07-01T08:13:34Z INFO worker.job id=814 queue=exports attempt=1 result=ok +2026-07-01T08:13:35Z WARN rate.limit tenant=t_815 remaining=17 window=60s +2026-07-01T08:13:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000816 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:37Z DEBUG cache.miss key=report:817 shard=3 ttl=900 +2026-07-01T08:13:38Z INFO worker.job id=818 queue=exports attempt=1 result=ok +2026-07-01T08:13:39Z WARN rate.limit tenant=t_819 remaining=17 window=60s +2026-07-01T08:13:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000820 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:41Z DEBUG cache.miss key=report:821 shard=3 ttl=900 +2026-07-01T08:13:42Z INFO worker.job id=822 queue=exports attempt=1 result=ok +2026-07-01T08:13:43Z WARN rate.limit tenant=t_823 remaining=17 window=60s +2026-07-01T08:13:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000824 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:45Z DEBUG cache.miss key=report:825 shard=3 ttl=900 +2026-07-01T08:13:46Z INFO worker.job id=826 queue=exports attempt=1 result=ok +2026-07-01T08:13:47Z WARN rate.limit tenant=t_827 remaining=17 window=60s +2026-07-01T08:13:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000828 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:49Z DEBUG cache.miss key=report:829 shard=3 ttl=900 +2026-07-01T08:13:50Z INFO worker.job id=830 queue=exports attempt=1 result=ok +2026-07-01T08:13:51Z WARN rate.limit tenant=t_831 remaining=17 window=60s +2026-07-01T08:13:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000832 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:53Z DEBUG cache.miss key=report:833 shard=3 ttl=900 +2026-07-01T08:13:54Z INFO worker.job id=834 queue=exports attempt=1 result=ok +2026-07-01T08:13:55Z WARN rate.limit tenant=t_835 remaining=17 window=60s +2026-07-01T08:13:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000836 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:13:57Z DEBUG cache.miss key=report:837 shard=3 ttl=900 +2026-07-01T08:13:58Z INFO worker.job id=838 queue=exports attempt=1 result=ok +2026-07-01T08:13:59Z WARN rate.limit tenant=t_839 remaining=17 window=60s +2026-07-01T08:14:00Z WARN pii.echo user_email=maria.gonzalez@example.com callback=(303) 555-0176 +2026-07-01T08:14:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000840 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:01Z DEBUG cache.miss key=report:841 shard=3 ttl=900 +2026-07-01T08:14:02Z INFO worker.job id=842 queue=exports attempt=1 result=ok +2026-07-01T08:14:03Z WARN rate.limit tenant=t_843 remaining=17 window=60s +2026-07-01T08:14:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000844 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:05Z DEBUG cache.miss key=report:845 shard=3 ttl=900 +2026-07-01T08:14:06Z INFO worker.job id=846 queue=exports attempt=1 result=ok +2026-07-01T08:14:07Z WARN rate.limit tenant=t_847 remaining=17 window=60s +2026-07-01T08:14:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000848 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:09Z DEBUG cache.miss key=report:849 shard=3 ttl=900 +2026-07-01T08:14:10Z INFO worker.job id=850 queue=exports attempt=1 result=ok +2026-07-01T08:14:11Z WARN rate.limit tenant=t_851 remaining=17 window=60s +2026-07-01T08:14:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000852 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:13Z DEBUG cache.miss key=report:853 shard=3 ttl=900 +2026-07-01T08:14:14Z INFO worker.job id=854 queue=exports attempt=1 result=ok +2026-07-01T08:14:15Z WARN rate.limit tenant=t_855 remaining=17 window=60s +2026-07-01T08:14:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000856 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:17Z DEBUG cache.miss key=report:857 shard=3 ttl=900 +2026-07-01T08:14:18Z INFO worker.job id=858 queue=exports attempt=1 result=ok +2026-07-01T08:14:19Z WARN rate.limit tenant=t_859 remaining=17 window=60s +2026-07-01T08:14:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000860 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:21Z DEBUG cache.miss key=report:861 shard=3 ttl=900 +2026-07-01T08:14:22Z INFO worker.job id=862 queue=exports attempt=1 result=ok +2026-07-01T08:14:23Z WARN rate.limit tenant=t_863 remaining=17 window=60s +2026-07-01T08:14:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000864 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:25Z DEBUG cache.miss key=report:865 shard=3 ttl=900 +2026-07-01T08:14:26Z INFO worker.job id=866 queue=exports attempt=1 result=ok +2026-07-01T08:14:27Z WARN rate.limit tenant=t_867 remaining=17 window=60s +2026-07-01T08:14:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000868 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:29Z DEBUG cache.miss key=report:869 shard=3 ttl=900 +2026-07-01T08:14:30Z INFO worker.job id=870 queue=exports attempt=1 result=ok +2026-07-01T08:14:31Z WARN rate.limit tenant=t_871 remaining=17 window=60s +2026-07-01T08:14:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000872 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:33Z DEBUG cache.miss key=report:873 shard=3 ttl=900 +2026-07-01T08:14:34Z INFO worker.job id=874 queue=exports attempt=1 result=ok +2026-07-01T08:14:35Z WARN rate.limit tenant=t_875 remaining=17 window=60s +2026-07-01T08:14:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000876 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:37Z DEBUG cache.miss key=report:877 shard=3 ttl=900 +2026-07-01T08:14:38Z INFO worker.job id=878 queue=exports attempt=1 result=ok +2026-07-01T08:14:39Z WARN rate.limit tenant=t_879 remaining=17 window=60s +2026-07-01T08:14:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000880 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:41Z DEBUG cache.miss key=report:881 shard=3 ttl=900 +2026-07-01T08:14:42Z INFO worker.job id=882 queue=exports attempt=1 result=ok +2026-07-01T08:14:43Z WARN rate.limit tenant=t_883 remaining=17 window=60s +2026-07-01T08:14:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000884 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:45Z DEBUG cache.miss key=report:885 shard=3 ttl=900 +2026-07-01T08:14:46Z INFO worker.job id=886 queue=exports attempt=1 result=ok +2026-07-01T08:14:47Z WARN rate.limit tenant=t_887 remaining=17 window=60s +2026-07-01T08:14:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000888 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:49Z DEBUG cache.miss key=report:889 shard=3 ttl=900 +2026-07-01T08:14:50Z INFO worker.job id=890 queue=exports attempt=1 result=ok +2026-07-01T08:14:51Z WARN rate.limit tenant=t_891 remaining=17 window=60s +2026-07-01T08:14:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000892 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:53Z DEBUG cache.miss key=report:893 shard=3 ttl=900 +2026-07-01T08:14:54Z INFO worker.job id=894 queue=exports attempt=1 result=ok +2026-07-01T08:14:55Z WARN rate.limit tenant=t_895 remaining=17 window=60s +2026-07-01T08:14:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000896 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:14:57Z DEBUG cache.miss key=report:897 shard=3 ttl=900 +2026-07-01T08:14:58Z INFO worker.job id=898 queue=exports attempt=1 result=ok +2026-07-01T08:14:59Z WARN rate.limit tenant=t_899 remaining=17 window=60s +2026-07-01T08:15:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000900 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:01Z DEBUG cache.miss key=report:901 shard=3 ttl=900 +2026-07-01T08:15:02Z INFO worker.job id=902 queue=exports attempt=1 result=ok +2026-07-01T08:15:03Z WARN rate.limit tenant=t_903 remaining=17 window=60s +2026-07-01T08:15:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000904 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:05Z DEBUG cache.miss key=report:905 shard=3 ttl=900 +2026-07-01T08:15:06Z INFO worker.job id=906 queue=exports attempt=1 result=ok +2026-07-01T08:15:07Z WARN rate.limit tenant=t_907 remaining=17 window=60s +2026-07-01T08:15:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000908 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:09Z DEBUG cache.miss key=report:909 shard=3 ttl=900 +2026-07-01T08:15:10Z INFO worker.job id=910 queue=exports attempt=1 result=ok +2026-07-01T08:15:11Z WARN rate.limit tenant=t_911 remaining=17 window=60s +2026-07-01T08:15:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000912 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:13Z DEBUG cache.miss key=report:913 shard=3 ttl=900 +2026-07-01T08:15:14Z INFO worker.job id=914 queue=exports attempt=1 result=ok +2026-07-01T08:15:15Z WARN rate.limit tenant=t_915 remaining=17 window=60s +2026-07-01T08:15:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000916 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:17Z DEBUG cache.miss key=report:917 shard=3 ttl=900 +2026-07-01T08:15:18Z INFO worker.job id=918 queue=exports attempt=1 result=ok +2026-07-01T08:15:19Z WARN rate.limit tenant=t_919 remaining=17 window=60s +2026-07-01T08:15:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000920 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:21Z DEBUG cache.miss key=report:921 shard=3 ttl=900 +2026-07-01T08:15:22Z INFO worker.job id=922 queue=exports attempt=1 result=ok +2026-07-01T08:15:23Z WARN rate.limit tenant=t_923 remaining=17 window=60s +2026-07-01T08:15:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000924 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:25Z DEBUG cache.miss key=report:925 shard=3 ttl=900 +2026-07-01T08:15:26Z INFO worker.job id=926 queue=exports attempt=1 result=ok +2026-07-01T08:15:27Z WARN rate.limit tenant=t_927 remaining=17 window=60s +2026-07-01T08:15:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000928 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:29Z DEBUG cache.miss key=report:929 shard=3 ttl=900 +2026-07-01T08:15:30Z INFO worker.job id=930 queue=exports attempt=1 result=ok +2026-07-01T08:15:31Z WARN rate.limit tenant=t_931 remaining=17 window=60s +2026-07-01T08:15:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000932 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:33Z DEBUG cache.miss key=report:933 shard=3 ttl=900 +2026-07-01T08:15:34Z INFO worker.job id=934 queue=exports attempt=1 result=ok +2026-07-01T08:15:35Z WARN rate.limit tenant=t_935 remaining=17 window=60s +2026-07-01T08:15:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000936 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:37Z DEBUG cache.miss key=report:937 shard=3 ttl=900 +2026-07-01T08:15:38Z INFO worker.job id=938 queue=exports attempt=1 result=ok +2026-07-01T08:15:39Z WARN rate.limit tenant=t_939 remaining=17 window=60s +2026-07-01T08:15:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000940 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:41Z DEBUG cache.miss key=report:941 shard=3 ttl=900 +2026-07-01T08:15:42Z INFO worker.job id=942 queue=exports attempt=1 result=ok +2026-07-01T08:15:43Z WARN rate.limit tenant=t_943 remaining=17 window=60s +2026-07-01T08:15:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000944 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:45Z DEBUG cache.miss key=report:945 shard=3 ttl=900 +2026-07-01T08:15:46Z INFO worker.job id=946 queue=exports attempt=1 result=ok +2026-07-01T08:15:47Z WARN rate.limit tenant=t_947 remaining=17 window=60s +2026-07-01T08:15:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000948 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:49Z DEBUG cache.miss key=report:949 shard=3 ttl=900 +2026-07-01T08:15:50Z INFO worker.job id=950 queue=exports attempt=1 result=ok +2026-07-01T08:15:51Z WARN rate.limit tenant=t_951 remaining=17 window=60s +2026-07-01T08:15:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000952 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:53Z DEBUG cache.miss key=report:953 shard=3 ttl=900 +2026-07-01T08:15:54Z INFO worker.job id=954 queue=exports attempt=1 result=ok +2026-07-01T08:15:55Z WARN rate.limit tenant=t_955 remaining=17 window=60s +2026-07-01T08:15:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000956 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:15:57Z DEBUG cache.miss key=report:957 shard=3 ttl=900 +2026-07-01T08:15:58Z INFO worker.job id=958 queue=exports attempt=1 result=ok +2026-07-01T08:15:59Z WARN rate.limit tenant=t_959 remaining=17 window=60s +2026-07-01T08:16:00Z WARN pii.echo user_email=alex.rivera@example.invalid callback=(212) 555-0142 +2026-07-01T08:16:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000960 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:01Z DEBUG cache.miss key=report:961 shard=3 ttl=900 +2026-07-01T08:16:02Z INFO worker.job id=962 queue=exports attempt=1 result=ok +2026-07-01T08:16:03Z WARN rate.limit tenant=t_963 remaining=17 window=60s +2026-07-01T08:16:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000964 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:05Z DEBUG cache.miss key=report:965 shard=3 ttl=900 +2026-07-01T08:16:06Z INFO worker.job id=966 queue=exports attempt=1 result=ok +2026-07-01T08:16:07Z WARN rate.limit tenant=t_967 remaining=17 window=60s +2026-07-01T08:16:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000968 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:09Z DEBUG cache.miss key=report:969 shard=3 ttl=900 +2026-07-01T08:16:10Z INFO worker.job id=970 queue=exports attempt=1 result=ok +2026-07-01T08:16:11Z WARN rate.limit tenant=t_971 remaining=17 window=60s +2026-07-01T08:16:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000972 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:13Z DEBUG cache.miss key=report:973 shard=3 ttl=900 +2026-07-01T08:16:14Z INFO worker.job id=974 queue=exports attempt=1 result=ok +2026-07-01T08:16:15Z WARN rate.limit tenant=t_975 remaining=17 window=60s +2026-07-01T08:16:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000976 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:17Z DEBUG cache.miss key=report:977 shard=3 ttl=900 +2026-07-01T08:16:18Z INFO worker.job id=978 queue=exports attempt=1 result=ok +2026-07-01T08:16:19Z WARN rate.limit tenant=t_979 remaining=17 window=60s +2026-07-01T08:16:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000980 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:21Z DEBUG cache.miss key=report:981 shard=3 ttl=900 +2026-07-01T08:16:22Z INFO worker.job id=982 queue=exports attempt=1 result=ok +2026-07-01T08:16:23Z WARN rate.limit tenant=t_983 remaining=17 window=60s +2026-07-01T08:16:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000984 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:25Z DEBUG cache.miss key=report:985 shard=3 ttl=900 +2026-07-01T08:16:26Z INFO worker.job id=986 queue=exports attempt=1 result=ok +2026-07-01T08:16:27Z WARN rate.limit tenant=t_987 remaining=17 window=60s +2026-07-01T08:16:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000988 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:29Z DEBUG cache.miss key=report:989 shard=3 ttl=900 +2026-07-01T08:16:30Z INFO worker.job id=990 queue=exports attempt=1 result=ok +2026-07-01T08:16:31Z WARN rate.limit tenant=t_991 remaining=17 window=60s +2026-07-01T08:16:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000992 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:33Z DEBUG cache.miss key=report:993 shard=3 ttl=900 +2026-07-01T08:16:34Z INFO worker.job id=994 queue=exports attempt=1 result=ok +2026-07-01T08:16:35Z WARN rate.limit tenant=t_995 remaining=17 window=60s +2026-07-01T08:16:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000000996 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:37Z DEBUG cache.miss key=report:997 shard=3 ttl=900 +2026-07-01T08:16:38Z INFO worker.job id=998 queue=exports attempt=1 result=ok +2026-07-01T08:16:39Z WARN rate.limit tenant=t_999 remaining=17 window=60s +2026-07-01T08:16:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001000 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:41Z DEBUG cache.miss key=report:1001 shard=3 ttl=900 +2026-07-01T08:16:42Z INFO worker.job id=1002 queue=exports attempt=1 result=ok +2026-07-01T08:16:43Z WARN rate.limit tenant=t_1003 remaining=17 window=60s +2026-07-01T08:16:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001004 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:45Z DEBUG cache.miss key=report:1005 shard=3 ttl=900 +2026-07-01T08:16:46Z INFO worker.job id=1006 queue=exports attempt=1 result=ok +2026-07-01T08:16:47Z WARN rate.limit tenant=t_1007 remaining=17 window=60s +2026-07-01T08:16:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001008 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:49Z DEBUG cache.miss key=report:1009 shard=3 ttl=900 +2026-07-01T08:16:50Z INFO worker.job id=1010 queue=exports attempt=1 result=ok +2026-07-01T08:16:51Z WARN rate.limit tenant=t_1011 remaining=17 window=60s +2026-07-01T08:16:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001012 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:53Z DEBUG cache.miss key=report:1013 shard=3 ttl=900 +2026-07-01T08:16:54Z INFO worker.job id=1014 queue=exports attempt=1 result=ok +2026-07-01T08:16:55Z WARN rate.limit tenant=t_1015 remaining=17 window=60s +2026-07-01T08:16:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001016 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:16:57Z DEBUG cache.miss key=report:1017 shard=3 ttl=900 +2026-07-01T08:16:58Z INFO worker.job id=1018 queue=exports attempt=1 result=ok +2026-07-01T08:16:59Z WARN rate.limit tenant=t_1019 remaining=17 window=60s +2026-07-01T08:17:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001020 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:01Z DEBUG cache.miss key=report:1021 shard=3 ttl=900 +2026-07-01T08:17:02Z INFO worker.job id=1022 queue=exports attempt=1 result=ok +2026-07-01T08:17:03Z WARN rate.limit tenant=t_1023 remaining=17 window=60s +2026-07-01T08:17:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001024 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:05Z DEBUG cache.miss key=report:1025 shard=3 ttl=900 +2026-07-01T08:17:06Z INFO worker.job id=1026 queue=exports attempt=1 result=ok +2026-07-01T08:17:07Z WARN rate.limit tenant=t_1027 remaining=17 window=60s +2026-07-01T08:17:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001028 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:09Z DEBUG cache.miss key=report:1029 shard=3 ttl=900 +2026-07-01T08:17:10Z INFO worker.job id=1030 queue=exports attempt=1 result=ok +2026-07-01T08:17:11Z WARN rate.limit tenant=t_1031 remaining=17 window=60s +2026-07-01T08:17:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001032 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:13Z DEBUG cache.miss key=report:1033 shard=3 ttl=900 +2026-07-01T08:17:14Z INFO worker.job id=1034 queue=exports attempt=1 result=ok +2026-07-01T08:17:15Z WARN rate.limit tenant=t_1035 remaining=17 window=60s +2026-07-01T08:17:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001036 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:17Z DEBUG cache.miss key=report:1037 shard=3 ttl=900 +2026-07-01T08:17:18Z INFO worker.job id=1038 queue=exports attempt=1 result=ok +2026-07-01T08:17:19Z WARN rate.limit tenant=t_1039 remaining=17 window=60s +2026-07-01T08:17:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001040 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:21Z DEBUG cache.miss key=report:1041 shard=3 ttl=900 +2026-07-01T08:17:22Z INFO worker.job id=1042 queue=exports attempt=1 result=ok +2026-07-01T08:17:23Z WARN rate.limit tenant=t_1043 remaining=17 window=60s +2026-07-01T08:17:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001044 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:25Z DEBUG cache.miss key=report:1045 shard=3 ttl=900 +2026-07-01T08:17:26Z INFO worker.job id=1046 queue=exports attempt=1 result=ok +2026-07-01T08:17:27Z WARN rate.limit tenant=t_1047 remaining=17 window=60s +2026-07-01T08:17:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001048 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:29Z DEBUG cache.miss key=report:1049 shard=3 ttl=900 +2026-07-01T08:17:30Z INFO worker.job id=1050 queue=exports attempt=1 result=ok +2026-07-01T08:17:31Z WARN rate.limit tenant=t_1051 remaining=17 window=60s +2026-07-01T08:17:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001052 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:33Z DEBUG cache.miss key=report:1053 shard=3 ttl=900 +2026-07-01T08:17:34Z INFO worker.job id=1054 queue=exports attempt=1 result=ok +2026-07-01T08:17:35Z WARN rate.limit tenant=t_1055 remaining=17 window=60s +2026-07-01T08:17:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001056 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:37Z DEBUG cache.miss key=report:1057 shard=3 ttl=900 +2026-07-01T08:17:38Z INFO worker.job id=1058 queue=exports attempt=1 result=ok +2026-07-01T08:17:39Z WARN rate.limit tenant=t_1059 remaining=17 window=60s +2026-07-01T08:17:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001060 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:41Z DEBUG cache.miss key=report:1061 shard=3 ttl=900 +2026-07-01T08:17:42Z INFO worker.job id=1062 queue=exports attempt=1 result=ok +2026-07-01T08:17:43Z WARN rate.limit tenant=t_1063 remaining=17 window=60s +2026-07-01T08:17:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001064 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:45Z DEBUG cache.miss key=report:1065 shard=3 ttl=900 +2026-07-01T08:17:46Z INFO worker.job id=1066 queue=exports attempt=1 result=ok +2026-07-01T08:17:47Z WARN rate.limit tenant=t_1067 remaining=17 window=60s +2026-07-01T08:17:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001068 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:49Z DEBUG cache.miss key=report:1069 shard=3 ttl=900 +2026-07-01T08:17:50Z INFO worker.job id=1070 queue=exports attempt=1 result=ok +2026-07-01T08:17:51Z WARN rate.limit tenant=t_1071 remaining=17 window=60s +2026-07-01T08:17:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001072 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:53Z DEBUG cache.miss key=report:1073 shard=3 ttl=900 +2026-07-01T08:17:54Z INFO worker.job id=1074 queue=exports attempt=1 result=ok +2026-07-01T08:17:55Z WARN rate.limit tenant=t_1075 remaining=17 window=60s +2026-07-01T08:17:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001076 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:17:57Z DEBUG cache.miss key=report:1077 shard=3 ttl=900 +2026-07-01T08:17:58Z INFO worker.job id=1078 queue=exports attempt=1 result=ok +2026-07-01T08:17:59Z WARN rate.limit tenant=t_1079 remaining=17 window=60s +2026-07-01T08:18:00Z WARN pii.echo user_email=j.chen@support.example.com callback=415-555-0188 +2026-07-01T08:18:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001080 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:01Z DEBUG cache.miss key=report:1081 shard=3 ttl=900 +2026-07-01T08:18:02Z INFO worker.job id=1082 queue=exports attempt=1 result=ok +2026-07-01T08:18:03Z WARN rate.limit tenant=t_1083 remaining=17 window=60s +2026-07-01T08:18:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001084 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:05Z DEBUG cache.miss key=report:1085 shard=3 ttl=900 +2026-07-01T08:18:06Z INFO worker.job id=1086 queue=exports attempt=1 result=ok +2026-07-01T08:18:07Z WARN rate.limit tenant=t_1087 remaining=17 window=60s +2026-07-01T08:18:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001088 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:09Z DEBUG cache.miss key=report:1089 shard=3 ttl=900 +2026-07-01T08:18:10Z INFO worker.job id=1090 queue=exports attempt=1 result=ok +2026-07-01T08:18:11Z WARN rate.limit tenant=t_1091 remaining=17 window=60s +2026-07-01T08:18:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001092 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:13Z DEBUG cache.miss key=report:1093 shard=3 ttl=900 +2026-07-01T08:18:14Z INFO worker.job id=1094 queue=exports attempt=1 result=ok +2026-07-01T08:18:15Z WARN rate.limit tenant=t_1095 remaining=17 window=60s +2026-07-01T08:18:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001096 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:17Z DEBUG cache.miss key=report:1097 shard=3 ttl=900 +2026-07-01T08:18:18Z INFO worker.job id=1098 queue=exports attempt=1 result=ok +2026-07-01T08:18:19Z WARN rate.limit tenant=t_1099 remaining=17 window=60s +2026-07-01T08:18:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001100 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:21Z DEBUG cache.miss key=report:1101 shard=3 ttl=900 +2026-07-01T08:18:22Z INFO worker.job id=1102 queue=exports attempt=1 result=ok +2026-07-01T08:18:23Z WARN rate.limit tenant=t_1103 remaining=17 window=60s +2026-07-01T08:18:24Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001104 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:25Z DEBUG cache.miss key=report:1105 shard=3 ttl=900 +2026-07-01T08:18:26Z INFO worker.job id=1106 queue=exports attempt=1 result=ok +2026-07-01T08:18:27Z WARN rate.limit tenant=t_1107 remaining=17 window=60s +2026-07-01T08:18:28Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001108 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:29Z DEBUG cache.miss key=report:1109 shard=3 ttl=900 +2026-07-01T08:18:30Z INFO worker.job id=1110 queue=exports attempt=1 result=ok +2026-07-01T08:18:31Z WARN rate.limit tenant=t_1111 remaining=17 window=60s +2026-07-01T08:18:32Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001112 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:33Z DEBUG cache.miss key=report:1113 shard=3 ttl=900 +2026-07-01T08:18:34Z INFO worker.job id=1114 queue=exports attempt=1 result=ok +2026-07-01T08:18:35Z WARN rate.limit tenant=t_1115 remaining=17 window=60s +2026-07-01T08:18:36Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001116 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:37Z DEBUG cache.miss key=report:1117 shard=3 ttl=900 +2026-07-01T08:18:38Z INFO worker.job id=1118 queue=exports attempt=1 result=ok +2026-07-01T08:18:39Z WARN rate.limit tenant=t_1119 remaining=17 window=60s +2026-07-01T08:18:40Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001120 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:41Z DEBUG cache.miss key=report:1121 shard=3 ttl=900 +2026-07-01T08:18:42Z INFO worker.job id=1122 queue=exports attempt=1 result=ok +2026-07-01T08:18:43Z WARN rate.limit tenant=t_1123 remaining=17 window=60s +2026-07-01T08:18:44Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001124 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:45Z DEBUG cache.miss key=report:1125 shard=3 ttl=900 +2026-07-01T08:18:46Z INFO worker.job id=1126 queue=exports attempt=1 result=ok +2026-07-01T08:18:47Z WARN rate.limit tenant=t_1127 remaining=17 window=60s +2026-07-01T08:18:48Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001128 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:49Z DEBUG cache.miss key=report:1129 shard=3 ttl=900 +2026-07-01T08:18:50Z INFO worker.job id=1130 queue=exports attempt=1 result=ok +2026-07-01T08:18:51Z WARN rate.limit tenant=t_1131 remaining=17 window=60s +2026-07-01T08:18:52Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001132 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:53Z DEBUG cache.miss key=report:1133 shard=3 ttl=900 +2026-07-01T08:18:54Z INFO worker.job id=1134 queue=exports attempt=1 result=ok +2026-07-01T08:18:55Z WARN rate.limit tenant=t_1135 remaining=17 window=60s +2026-07-01T08:18:56Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001136 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:18:57Z DEBUG cache.miss key=report:1137 shard=3 ttl=900 +2026-07-01T08:18:58Z INFO worker.job id=1138 queue=exports attempt=1 result=ok +2026-07-01T08:18:59Z WARN rate.limit tenant=t_1139 remaining=17 window=60s +2026-07-01T08:19:00Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001140 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:19:01Z DEBUG cache.miss key=report:1141 shard=3 ttl=900 +2026-07-01T08:19:02Z INFO worker.job id=1142 queue=exports attempt=1 result=ok +2026-07-01T08:19:03Z WARN rate.limit tenant=t_1143 remaining=17 window=60s +2026-07-01T08:19:04Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001144 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:19:05Z DEBUG cache.miss key=report:1145 shard=3 ttl=900 +2026-07-01T08:19:06Z INFO worker.job id=1146 queue=exports attempt=1 result=ok +2026-07-01T08:19:07Z WARN rate.limit tenant=t_1147 remaining=17 window=60s +2026-07-01T08:19:08Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001148 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:19:09Z DEBUG cache.miss key=report:1149 shard=3 ttl=900 +2026-07-01T08:19:10Z INFO worker.job id=1150 queue=exports attempt=1 result=ok +2026-07-01T08:19:11Z WARN rate.limit tenant=t_1151 remaining=17 window=60s +2026-07-01T08:19:12Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001152 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:19:13Z DEBUG cache.miss key=report:1153 shard=3 ttl=900 +2026-07-01T08:19:14Z INFO worker.job id=1154 queue=exports attempt=1 result=ok +2026-07-01T08:19:15Z WARN rate.limit tenant=t_1155 remaining=17 window=60s +2026-07-01T08:19:16Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001156 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:19:17Z DEBUG cache.miss key=report:1157 shard=3 ttl=900 +2026-07-01T08:19:18Z INFO worker.job id=1158 queue=exports attempt=1 result=ok +2026-07-01T08:19:19Z WARN rate.limit tenant=t_1159 remaining=17 window=60s +2026-07-01T08:19:20Z INFO api.request id=9f8c2d1e-4b7a-4c3e-9d2f-000000001160 route=/v1/reports status=200 dur_ms=41 +2026-07-01T08:19:21Z DEBUG cache.miss key=report:1161 shard=3 ttl=900 diff --git a/benchmarks/payloads/core_10kb_mixed.txt b/benchmarks/payloads/core_10kb_mixed.txt new file mode 100644 index 00000000..a96912a6 --- /dev/null +++ b/benchmarks/payloads/core_10kb_mixed.txt @@ -0,0 +1,31 @@ +Section 1 +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. +Contact: alex.rivera@example.invalid or (212) 555-0142. + +Section 2 +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. +Payment reference: card ending 4111-1111-1111-1111, verification 078-05-1120. + +Section 3 +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. +Contact: billing-team@example.invalid or +1 646 555 0123. + +Section 4 +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. +Section 5 +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. +Contact: alex.rivera@example.invalid or (212) 555-0142. + +Section 6 +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. +Payment reference: card ending 4111-1111-1111-1111, verification 078-05-1120. + +Section 7 +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. +Contact: billing-team@example.invalid or +1 646 555 0123. + +Section 8 +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain diff --git a/benchmarks/payloads/core_1kb_dense.txt b/benchmarks/payloads/core_1kb_dense.txt new file mode 100644 index 00000000..d51e6f20 --- /dev/null +++ b/benchmarks/payloads/core_1kb_dense.txt @@ -0,0 +1,8 @@ +Support ticket #48213 — account verification + +Customer alex.rivera@example.invalid called from (212) 555-0142 about a failed payment on card 4242424242424242. Identity was confirmed against SSN 219-09-9999. + +Escalated to j.chen@support.example.com; callback scheduled at 415-555-0188. A retry was attempted with backup card 4111-1111-1111-1111, which also declined. Secondary account holder (SSN 078-05-1120) authorized the change. + +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. +The quarterly review covered onboarding latency, retention cohorts, and the migration of the billing reconciliation job to the new scheduler. Action items were assigned to the platform team with a two-week horizon. No regressions were observed in the canary environment during the rollout window, and error budgets remain within policy for all tier-one services. Resolution: card replaced, customer notified by email, ticket closed. diff --git a/benchmarks/payloads/hook_pretooluse.json b/benchmarks/payloads/hook_pretooluse.json new file mode 100644 index 00000000..f4456534 --- /dev/null +++ b/benchmarks/payloads/hook_pretooluse.json @@ -0,0 +1,8 @@ +{ + "hook_event_name": "PreToolUse", + "tool_name": "Bash", + "tool_input": { + "command": "curl -X POST https://api.example.invalid/v1/customers -d 'email=alex.rivera@example.invalid&phone=(212) 555-0142&card=4242424242424242'", + "description": "Create customer record via API" + } +} diff --git a/benchmarks/payloads/manifest.json b/benchmarks/payloads/manifest.json new file mode 100644 index 00000000..5e5ae29e --- /dev/null +++ b/benchmarks/payloads/manifest.json @@ -0,0 +1,31 @@ +{ + "entity_types": ["EMAIL", "PHONE", "CREDIT_CARD", "SSN"], + "note": "All PII values are synthetic test values (reserved domains, 555-01XX phones, public test cards, SSA example SSNs). expected_counts pins what the regex engine must detect before timing begins.", + "payloads": { + "core_100kb_sparse.txt": { + "bytes": 100000, + "expected_counts": { + "EMAIL": 9, + "PHONE": 9 + } + }, + "core_10kb_mixed.txt": { + "bytes": 10241, + "expected_counts": { + "CREDIT_CARD": 2, + "EMAIL": 4, + "PHONE": 4, + "SSN": 2 + } + }, + "core_1kb_dense.txt": { + "bytes": 1232, + "expected_counts": { + "CREDIT_CARD": 2, + "EMAIL": 2, + "PHONE": 2, + "SSN": 2 + } + } + } +} diff --git a/benchmarks/requirements.txt b/benchmarks/requirements.txt new file mode 100644 index 00000000..6b84218f --- /dev/null +++ b/benchmarks/requirements.txt @@ -0,0 +1,9 @@ +# Dependencies for the full benchmark suite. The core and hook suites need +# only datafog itself; these extras enable the guardrail and comparison +# suites. Kept out of the package's install_requires/extras on purpose. +litellm>=1.53 +fastapi>=0.115 +presidio-analyzer>=2.2 +spacy>=3.7 +# Also required once, for the spacy/presidio comparison suites: +# python -m spacy download en_core_web_sm diff --git a/benchmarks/run.py b/benchmarks/run.py new file mode 100644 index 00000000..0f9d0a7c --- /dev/null +++ b/benchmarks/run.py @@ -0,0 +1,176 @@ +"""DataFog benchmark suite — reproduce every published performance number. + +Usage:: + + python benchmarks/run.py # all suites (skips missing deps) + python benchmarks/run.py --suite core,hook # subset + python benchmarks/run.py --json results.json # machine-readable output + python benchmarks/run.py --quick # fewer repeats, smaller payloads + +Suites: core, guardrail, hook, spacy, presidio. See benchmarks/README.md +for methodology and how each number maps to the claims in the project +README and on datafog.ai. +""" + +from __future__ import annotations + +import argparse +import json +import platform +import subprocess +import sys +from datetime import datetime, timezone +from importlib import metadata +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).parent)) + +from _suites import SUITES, SuiteUnavailable # noqa: E402 +from _timing import Measurement # noqa: E402 + + +def _cpu_name() -> str: + if sys.platform == "darwin": + try: + return subprocess.run( + ["sysctl", "-n", "machdep.cpu.brand_string"], + capture_output=True, + text=True, + check=True, + ).stdout.strip() + except (OSError, subprocess.CalledProcessError): + pass + return platform.processor() or platform.machine() + + +def _package_versions() -> dict[str, str]: + versions = {} + for package in ("datafog", "litellm", "presidio-analyzer", "spacy", "pydantic"): + try: + versions[package] = metadata.version(package) + except metadata.PackageNotFoundError: + versions[package] = "not installed" + return versions + + +def environment() -> dict[str, object]: + return { + "timestamp": datetime.now(timezone.utc).isoformat(timespec="seconds"), + "platform": platform.platform(), + "cpu": _cpu_name(), + "python": platform.python_version(), + "packages": _package_versions(), + } + + +def _fmt_value(value: float, unit: str) -> str: + if unit == "us/op": + if value >= 100_000: + return f"{value / 1e6:8.2f} s" + if value >= 1_000: + return f"{value / 1e3:8.2f} ms" + return f"{value:8.1f} µs" + return f"{value:8.1f} ms" + + +def _fmt_meta(meta: dict[str, object]) -> str: + parts = [] + for key in ( + "throughput_mb_s", + "regex_speedup_x", + "datafog_speedup_x", + "p90", + "detected", + "note", + ): + if key in meta: + value = meta[key] + if key == "throughput_mb_s": + parts.append(f"{value} MB/s") + elif key.endswith("_speedup_x"): + parts.append(f"{key.split('_speedup')[0]} is {value}x faster") + elif key == "p90": + parts.append(f"p90 {value:.1f} ms") + else: + parts.append(f"{key}: {value}") + return "; ".join(str(p) for p in parts) + + +def print_report(results_by_suite: dict[str, list[Measurement]]) -> None: + for suite, measurements in results_by_suite.items(): + print(f"\n## {suite}") + width = max(len(m.name) for m in measurements) + for m in measurements: + spread = ( + f"±{_fmt_value(m.stdev, m.unit).strip():>10}" + if m.repeats > 1 + else " " * 11 + ) + line = ( + f" {m.name:<{width}} median {_fmt_value(m.median, m.unit)} " + f"{spread} ({m.repeats}x{m.inner_loops} runs)" + ) + extra = _fmt_meta(m.meta) + print(line + (f"\n {'':<{width}} {extra}" if extra else "")) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument( + "--suite", + default="all", + help=f"comma-separated subset of: {', '.join(SUITES)} (default: all)", + ) + parser.add_argument( + "--json", type=Path, default=None, help="write results to a JSON file" + ) + parser.add_argument( + "--quick", action="store_true", help="fewer repeats, skip large payloads" + ) + args = parser.parse_args() + + if args.suite == "all": + selected = list(SUITES) + else: + selected = [s.strip() for s in args.suite.split(",") if s.strip()] + unknown = [s for s in selected if s not in SUITES] + if unknown: + parser.error( + f"unknown suite(s): {', '.join(unknown)}; choose from {', '.join(SUITES)}" + ) + + env = environment() + print("DataFog benchmark suite") + print(f" {env['cpu']} | {env['platform']} | Python {env['python']}") + print(" " + ", ".join(f"{k} {v}" for k, v in env["packages"].items())) + + results_by_suite: dict[str, list[Measurement]] = {} + skipped: dict[str, str] = {} + for suite in selected: + print(f"\nrunning suite: {suite} ...", flush=True) + try: + results_by_suite[suite] = SUITES[suite](args.quick) + except SuiteUnavailable as exc: + skipped[suite] = str(exc) + print(f" skipped: {exc}") + + print_report(results_by_suite) + if skipped: + print("\nskipped suites:") + for suite, reason in skipped.items(): + print(f" {suite}: {reason}") + + if args.json: + payload = { + "environment": env, + "results": [m.to_dict() for ms in results_by_suite.values() for m in ms], + "skipped": skipped, + } + args.json.write_text(json.dumps(payload, indent=2) + "\n") + print(f"\nresults written to {args.json}") + + return 0 if results_by_suite else 1 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/datafog/__about__.py b/datafog/__about__.py index 8355eb42..c4f16a13 100644 --- a/datafog/__about__.py +++ b/datafog/__about__.py @@ -1 +1 @@ -__version__ = "4.7.0" +__version__ = "4.7.0a1" diff --git a/datafog/__init__.py b/datafog/__init__.py index f1bd8daf..7236ac10 100644 --- a/datafog/__init__.py +++ b/datafog/__init__.py @@ -1,7 +1,8 @@ """ DataFog: Lightning-fast PII detection and anonymization library. -Core package provides regex-based PII detection with 190x performance advantage. +Core package provides regex-based PII detection, 100x+ faster than NER-based +detection on identical payloads (reproduce: python benchmarks/run.py). Optional extras available for advanced features: - pip install datafog[nlp] - for spaCy integration - pip install datafog[ocr] - for image/OCR processing diff --git a/datafog/__init___lean.py b/datafog/__init___lean.py index 5c25f9bc..3f1165b8 100644 --- a/datafog/__init___lean.py +++ b/datafog/__init___lean.py @@ -7,7 +7,8 @@ DataFog: Lightning-fast PII detection and anonymization library. -Core package provides regex-based PII detection with 190x performance advantage. +Core package provides regex-based PII detection, 100x+ faster than NER-based +detection on identical payloads (reproduce: python benchmarks/run.py). Optional extras available for advanced features: - pip install datafog[nlp] - for spaCy integration - pip install datafog[ocr] - for image/OCR processing diff --git a/datafog/engine.py b/datafog/engine.py index 46ed72a8..419558e9 100644 --- a/datafog/engine.py +++ b/datafog/engine.py @@ -173,6 +173,7 @@ def _suppress_overlapping_entities(entities: list[Entity]) -> list[Entity]: key=lambda item: ( -_entity_length(item), -ENTITY_TYPE_PRIORITY.get(item.type, 0), + -item.confidence, item.start, item.end, item.type, @@ -465,7 +466,18 @@ def redact( entities: list[Entity], strategy: str = "token", ) -> RedactResult: - """Redact PII entities from text.""" + """Redact PII entities from text. + + Callers may pass arbitrary entity lists (not just ``scan()`` output), so + overlapping or duplicate spans are resolved here: for each overlapping + group only one span is redacted — the longest, breaking ties by entity + type priority and then confidence. Suppressed spans are omitted from + ``RedactResult.entities`` and ``mapping``. + + ``mapping`` for the ``mask`` strategy is keyed by per-type indexed keys + (``[EMAIL_MASK_1]``) rather than the mask string, because distinct + same-length values produce identical masks and would collide. + """ if not isinstance(text, str): raise TypeError("text must be a string") if strategy not in {"token", "mask", "hash", "pseudonymize"}: @@ -481,14 +493,22 @@ def redact( for entity in entities if 0 <= entity.start < entity.end <= len(text) and entity.text ] - valid_entities = sorted( - valid_entities, key=lambda e: (e.start, e.end), reverse=True - ) + # Returns non-overlapping spans sorted by document position. + valid_entities = _suppress_overlapping_entities(valid_entities) + # Assign replacements in document order so the per-type counters used by + # the token/pseudonymize strategies number the first occurrence _1; spans + # are applied right-to-left below so earlier offsets stay valid. + replacements: list[tuple[Entity, str]] = [] for entity in valid_entities: - original = redacted_text[entity.start : entity.end] + original = text[entity.start : entity.end] + mapping_key: Optional[str] = None if strategy == "mask": replacement = "*" * max(len(original), 1) + # Distinct same-length values share a mask, so the mask string + # cannot key the mapping without collisions. + counters[entity.type] = counters.get(entity.type, 0) + 1 + mapping_key = f"[{entity.type}_MASK_{counters[entity.type]}]" elif strategy == "hash": digest = hashlib.sha256(original.encode("utf-8")).hexdigest()[:12] replacement = f"[{entity.type}_{digest}]" @@ -504,10 +524,13 @@ def redact( counters[entity.type] = counters.get(entity.type, 0) + 1 replacement = f"[{entity.type}_{counters[entity.type]}]" + replacements.append((entity, replacement)) + mapping[mapping_key if mapping_key is not None else replacement] = original + + for entity, replacement in reversed(replacements): redacted_text = ( redacted_text[: entity.start] + replacement + redacted_text[entity.end :] ) - mapping[replacement] = original return RedactResult( redacted_text=redacted_text, diff --git a/docs/roadmap.rst b/docs/roadmap.rst index 1ea8bbc4..d350801b 100644 --- a/docs/roadmap.rst +++ b/docs/roadmap.rst @@ -2,162 +2,165 @@ Release Roadmap ================ -This roadmap outlines the evolution of DataFog from a monolithic package -to a lightweight, modular architecture with optional extras. +Where DataFog is today (4.7.x) and where it is going (v5.0.0). The 4.x +line delivered the lightweight-core architecture and, from 4.6.0 on, an +offline PII firewall for AI agents and gateways. The v5 cycle turns that +foundation into the fastest, easiest offline PII firewall for AI apps, +logs, and datasets. .. contents:: Table of Contents :local: :depth: 1 -v4.4.0 - Python 3.13 and v5 Migration Bridge --------------------------------------------- +Current status — 4.7.x +---------------------- -Before v5.0.0, DataFog should ship a focused v4.4.0 bridge release. The -purpose is to give users a concrete compatibility win while introducing the -v5 direction gently. +DataFog ``4.7.0`` is the current stable release (July 2026). Patch +releases on the 4.7.x line focus on detection precision — reducing +SSN/phone false positives surfaced by real-world agent traffic — while +v5.0.0 is prepared. -v4.4.0 should focus on: +Every published performance number is reproducible with one command: +``python benchmarks/run.py`` (see ``benchmarks/`` for methodology, +pinned payloads, and comparisons against Presidio and spaCy NER). -* Python 3.13 support for the core SDK and CLI. -* Dependency validation for optional profiles without blocking core/CLI. -* v5-style preview APIs where they can land safely. -* Targeted deprecation warnings with no warnings on import. -* Migration docs and release notes that announce the v5 path. +✅ 4.1.0 — Lightweight Core (Released) +-------------------------------------- -Scope artifacts: +A major architectural shift to a lightweight core with optional extras: -* :doc:`v44-bridge-release` -* :doc:`v45-release-readiness` +* **Core package size reduced** from ~8MB to <2MB +* **Dependency splitting** into optional extras (nlp, ocr, distributed, etc.) +* **Simple API** with ``detect()`` and ``process()`` functions +* **Graceful degradation** when optional dependencies are not installed +* **100x+ performance advantage** over NER-based detection, now + reproducible with one command (``python benchmarks/run.py``) -v5.0.0 - Offline PII Firewall for AI Apps ------------------------------------------ +.. code-block:: bash -The v5.0.0 release is scoped around a sharper adoption wedge: + pip install datafog # lightweight core (regex engine only) + pip install datafog[nlp] # with spaCy + pip install datafog[ocr] # with OCR + pip install datafog[all] # everything - DataFog should be the fastest, easiest offline PII firewall for AI apps, - logs, and datasets. +✅ 4.2.x – 4.3.x — Engines and Operations (Released) +---------------------------------------------------- -The release should prioritize trust and time-to-first-value over broad -enterprise surface area. The first path should be a core install, a simple -top-level API, no network surprises, and copy-pasteable workflows for LLM -prompts/outputs, logs, JSONL datasets, and CI checks. +* **GLiNER engine** (4.2.0): modern NER via the ``nlp-advanced`` extra, + with a ``smart`` cascade mode. +* **Anonymous opt-out telemetry** (4.3.0) and consolidated CI workflows. +* Sample notebooks and performance-regression fixes. -Scope artifacts: +✅ 4.4.0 — Python 3.13 and v5 Migration Bridge (Released) +---------------------------------------------------------- -* :doc:`v5-product-brief` -* :doc:`v5-compatibility-matrix` -* :doc:`v5-cut-line` +The bridge release that introduced the v5 direction gently: -v5.0.0 must focus on: - -* Stable top-level APIs: ``scan``, ``redact``, ``protect``, and ``restore``. -* Privacy-safe defaults: no default network behavior, no runtime package - installation, and no implicit model downloads. -* Policy-based redaction with presets for LLMs, logs, strict workflows, and - datasets. -* Reversible token sessions that are explicit and opt-in. -* LLM guardrails, including sync, async, and streaming protection. -* CLI workflows for stdin, files, directories, CSV, JSONL, machine-readable - output, and CI-friendly exit codes. -* Custom recognizers and stronger structured detection for app/log/secrets - data. -* Modern packaging and release gates for install profiles, no-network - behavior, import time, wheel size, accuracy, coverage, and benchmarks. +* Python 3.13 support for the core SDK and CLI. +* Dependency validation for optional profiles without blocking core/CLI. +* v5-style preview APIs where they could land safely. +* Targeted deprecation warnings with no warnings on import. +* Migration docs and release notes announcing the v5 path. -Deferred to v5.1+: +Scope artifacts: :doc:`v44-bridge-release` -* OCR overhaul. -* Spark overhaul. -* Cloud DLP integrations. -* Enterprise dashboards and analytics. -* Broad multilingual model tuning. -* Large Presidio-style framework expansion. +✅ 4.5.0 — Lightweight Text PII Screening (Released July 2026) +--------------------------------------------------------------- -✅ 4.1.0 (Released) --------------------- -The ``4.1.0`` release represents a major architectural shift to a lightweight -core with optional extras. **Key achievements:** +A focus release that made the core package easier to install, reason +about, test, and use: -**Lightweight Architecture** +* Core text scanning, redaction, and guardrail helpers that stay + dependency-light by default (``scan``, ``redact``, ``protect`` previews). +* Clear install-profile documentation for core, NLP, OCR, Spark, CLI, and + web surfaces. +* German PII locale support. +* Documentation cleanup separating the current package story from + historical planning material. -* **Core package size reduced** from ~8MB to <2MB -* **Dependency splitting** into optional extras (nlp, ocr, distributed, etc.) -* **Simple API** with ``detect()`` and ``process()`` functions -* **Graceful degradation** when optional dependencies not installed +Scope artifacts: :doc:`v45-release-readiness` -**Performance Validation** +✅ 4.6.0 — Agent & Gateway Firewall (Released July 2026) +--------------------------------------------------------- -* **190x performance advantage** over spaCy validated with fair benchmarks -* **Independent benchmark scripts** for transparent performance claims -* **Regex engine optimization** maintaining sub-3ms processing times +Two ready-made enforcement points that catch PII at the moment it would +leave the machine — offline, with matched values never echoed into logs +or transcripts: -**Developer Experience** +* **Claude Code hook** (``datafog-hook``): gates agent tool calls + (shell commands, web requests, file writes, MCP tools) and warns the + model when prompts or tool results carry PII. ~70–90ms per invocation + including process startup; also installable as the + `Claude Code plugin `_. +* **LiteLLM guardrail** (``DataFogGuardrail``): redacts or blocks PII in + requests and responses at the gateway, for any LiteLLM-proxied + provider. In-process, ~40µs per message scanned — no sidecar service. -* **Streamlined CI/CD** with unified workflows and pre-commit integration -* **Auto-fix PRs** for formatting issues -* **Comprehensive testing** including dependency isolation tests +Both default to the high-precision entity set (``EMAIL``, ``PHONE``, +``CREDIT_CARD``, ``SSN``); noisier types are opt-in. -**Critical Stability Fixes (December 2024)** +✅ 4.7.0 — Allowlists and Typing (Released July 2026) +------------------------------------------------------ -* **CI/CD stabilization** with 87% test success rate (156/180 tests passing) -* **Structured output bug resolution** for multi-chunk text processing -* **Conditional testing architecture** preserving lean design while enabling full feature testing -* **Mock fixture corrections** for proper service isolation in tests -* **Benchmark test validation** ensuring performance claims remain verifiable +* **Allowlist support**: ``allowlist=[...]`` for exact values and + ``allowlist_patterns=[...]`` for full-match regexes (ReDoS-hardened), + available in the API and both firewall adapters. +* **Presidio-style entity aliases** (``EMAIL_ADDRESS``, ``PHONE_NUMBER``, + ``US_SSN``) for easy migration. +* **py.typed** marker for type-checker support. -**Installation Options** +4.7.x patch releases: SSN and phone precision fixes (hex-string and +no-dash false positives). -.. code-block:: bash +v5.0.0 — Offline PII Firewall for AI Apps (In Progress) +-------------------------------------------------------- + +The v5.0.0 release is scoped around a sharper adoption wedge: + + DataFog should be the fastest, easiest offline PII firewall for AI apps, + logs, and datasets. + +The release prioritizes trust and time-to-first-value over broad +enterprise surface area: a core install, a simple top-level API, no +network surprises, and copy-pasteable workflows for LLM prompts/outputs, +logs, JSONL datasets, and CI checks. + +Themes: + +* **Precision**: a zero-dependency validation pass on every hit (Luhn, + IBAN mod-97, SSN structure rules, IP plausibility), confidence scores + on findings, and strictness presets (``strict`` / ``balanced`` / + ``lenient``) — with precision/recall benchmarks published alongside the + speed benchmarks. +* **EU language & entity coverage**: generalize 4.5's German support into + a locale pack system (new languages become data plus tests, not code + changes), starting with locale-independent EU entities (IBAN, VAT IDs, + national ID formats). +* **A leaner package**: core stays pydantic-only; the Donut/transformers + OCR path is removed (pytesseract remains), the PySpark wrapper becomes + a documented recipe, legacy duplicate modules are deleted, and CI + enforces wheel-size and import-time budgets. +* **Built for pipelines**: pure, stateless, thread-safe ``scan``/``redact`` + with batch/iterator APIs, plus recipes for Kafka consumers, + Vector/Fluent Bit transforms, and OpenTelemetry collector processors. +* **Vault-friendly anonymization**: deterministic tokenization with + exportable mappings and format-preserving pseudonymization. + +API direction: ``scan()``, ``redact()``, ``protect()``, and ``restore()`` +become the primary documented API; legacy ``detect()``/``process()`` keep +working as compatibility shims throughout the v5.x line. - # Lightweight core (regex engine only) - pip install datafog - - # With spaCy for advanced NLP - pip install datafog[nlp] - - # With OCR capabilities - pip install datafog[ocr] - - # Full functionality - pip install datafog[all] - -4.2.x – 4.4.x --------------- -Subsequent minor releases will focus on: - -* **Enhanced regex patterns** for new entity types -* **Performance optimizations** maintaining 150x+ speedup advantage -* **Additional anonymization methods** (advanced hashing, format-preserving) -* **Improved OCR accuracy** with preprocessing pipelines -* **Extended CLI capabilities** for batch processing - -All features will remain backward compatible with the lightweight architecture. - -4.5.0 ------- -Version ``4.5.0`` is a focus release for lightweight text PII screening. It -should make the core package easier to install, reason about, test, and use -before larger v5 middleware work. - -4.5.0 should focus on: - -* Core text scanning, redaction, and guardrail helpers that stay dependency - light by default. -* Clear install-profile documentation for core, NLP, OCR, Spark, CLI, and web - surfaces. -* OCR and Spark as supported optional surfaces, not the main 4.5 adoption path. -* Documentation cleanup so users and contributors can find the current package - story without reading historical planning material first. -* German PII regex support if the external PR passes review and does not - compromise core precision. - -Deferred beyond 4.5.0: - -* Full middleware adapters for Sentry, OpenTelemetry, logging frameworks, or - cloud DLP services. -* OCR architecture overhaul. -* Spark architecture overhaul. +Deferred to v5.1+: + +* OCR overhaul. +* Spark overhaul. +* Cloud DLP integrations. * Enterprise dashboards and analytics. +* Broad multilingual model tuning. +* Large Presidio-style framework expansion. -The lightweight core remains the first path; optional surfaces should stay -explicit and isolated from default import, scan, redact, and guardrail usage. +Scope artifacts: + +* :doc:`v5-product-brief` +* :doc:`v5-compatibility-matrix` +* :doc:`v5-cut-line` diff --git a/setup.py b/setup.py index 078a91bf..1da9ebe3 100644 --- a/setup.py +++ b/setup.py @@ -110,7 +110,7 @@ version=version, author="Sid Mohan", author_email="sid@datafog.ai", - description="Lightning-fast PII detection and anonymization library with 190x performance advantage", + description="Lightning-fast PII detection and anonymization library, 100x+ faster than NER-based detection (see benchmarks/)", long_description=long_description, long_description_content_type="text/markdown", packages=find_packages(exclude=["tests", "tests.*"]), diff --git a/setup_lean.py b/setup_lean.py index ece73a1f..3d8296bb 100644 --- a/setup_lean.py +++ b/setup_lean.py @@ -78,7 +78,7 @@ version=__version__, author="Sid Mohan", author_email="sid@datafog.ai", - description="Lightning-fast PII detection and anonymization library with 190x performance advantage", + description="Lightning-fast PII detection and anonymization library, 100x+ faster than NER-based detection (see benchmarks/)", long_description=long_description, long_description_content_type="text/markdown", packages=find_packages(), diff --git a/tests/test_engine_api.py b/tests/test_engine_api.py index fdec81fe..2a43e147 100644 --- a/tests/test_engine_api.py +++ b/tests/test_engine_api.py @@ -56,6 +56,143 @@ def test_redact_strategies(strategy: str) -> None: assert result.mapping +def _entity_at(text: str, value: str, entity_type: str = "EMAIL") -> Entity: + start = text.index(value) + return Entity( + type=entity_type, + text=value, + start=start, + end=start + len(value), + confidence=1.0, + engine="regex", + ) + + +def test_redact_token_numbering_follows_document_order() -> None: + text = "first alpha@example.com then beta@example.com end" + entities = [ + _entity_at(text, "alpha@example.com"), + _entity_at(text, "beta@example.com"), + ] + + result = redact(text=text, entities=entities, strategy="token") + + assert result.redacted_text == "first [EMAIL_1] then [EMAIL_2] end" + assert result.mapping == { + "[EMAIL_1]": "alpha@example.com", + "[EMAIL_2]": "beta@example.com", + } + + +def test_redact_pseudonymize_numbering_follows_document_order() -> None: + text = "first alpha@example.com then beta@example.com end" + entities = [ + _entity_at(text, "alpha@example.com"), + _entity_at(text, "beta@example.com"), + ] + + result = redact(text=text, entities=entities, strategy="pseudonymize") + + assert result.redacted_text == "first [EMAIL_PSEUDO_1] then [EMAIL_PSEUDO_2] end" + assert result.mapping == { + "[EMAIL_PSEUDO_1]": "alpha@example.com", + "[EMAIL_PSEUDO_2]": "beta@example.com", + } + + +def _span( + text: str, + start: int, + end: int, + entity_type: str = "PHONE", + confidence: float = 1.0, + engine: str = "regex", +) -> Entity: + return Entity( + type=entity_type, + text=text[start:end], + start=start, + end=end, + confidence=confidence, + engine=engine, + ) + + +@pytest.mark.parametrize("strategy", ["token", "mask", "hash", "pseudonymize"]) +def test_redact_overlapping_spans_match_longest_only(strategy: str) -> None: + text = "call 555-000-1111 now" + full = _span(text, 5, 17) + suffix = _span(text, 9, 17) + + result = redact(text=text, entities=[full, suffix], strategy=strategy) + expected = redact(text=text, entities=[full], strategy=strategy) + + assert result.redacted_text == expected.redacted_text + assert result.mapping == expected.mapping + assert result.entities == [full] + + +@pytest.mark.parametrize("strategy", ["token", "mask", "hash", "pseudonymize"]) +def test_redact_nested_spans_keep_outer(strategy: str) -> None: + text = "mail alice@example.com today" + outer = _span(text, 5, 22, entity_type="EMAIL") + inner = _span(text, 11, 18, entity_type="EMAIL") + + result = redact(text=text, entities=[inner, outer], strategy=strategy) + expected = redact(text=text, entities=[outer], strategy=strategy) + + assert result.redacted_text == expected.redacted_text + assert result.mapping == expected.mapping + assert result.entities == [outer] + + +def test_redact_overlapping_spans_produce_clean_token_output() -> None: + text = "call 555-000-1111 now" + entities = [_span(text, 5, 17), _span(text, 9, 17)] + + result = redact(text=text, entities=entities, strategy="token") + + assert result.redacted_text == "call [PHONE_1] now" + assert result.mapping == {"[PHONE_1]": text[5:17]} + + +def test_redact_duplicate_spans_applied_once() -> None: + text = "mail alice@example.com today" + entity = _entity_at(text, "alice@example.com") + + result = redact(text=text, entities=[entity, entity], strategy="token") + + assert result.redacted_text == "mail [EMAIL_1] today" + assert result.entities == [entity] + + +def test_redact_overlap_tiebreak_prefers_higher_confidence() -> None: + text = "Acme Corporation announced" + org = _span(text, 0, 16, entity_type="ORGANIZATION", confidence=0.7, engine="spacy") + person = _span(text, 0, 16, entity_type="PERSON", confidence=0.9, engine="gliner") + + result = redact(text=text, entities=[org, person], strategy="token") + + assert result.redacted_text == "[PERSON_1] announced" + assert result.entities == [person] + + +def test_redact_mask_mapping_distinguishes_same_length_values() -> None: + text = "a alice@example.com b bobby@example.com c" + entities = [ + _entity_at(text, "alice@example.com"), + _entity_at(text, "bobby@example.com"), + ] + + result = redact(text=text, entities=entities, strategy="mask") + + assert result.redacted_text == "a " + "*" * 17 + " b " + "*" * 17 + " c" + assert result.mapping == { + "[EMAIL_MASK_1]": "alice@example.com", + "[EMAIL_MASK_2]": "bobby@example.com", + } + + def test_redact_invalid_strategy_raises_value_error() -> None: with pytest.raises(ValueError, match="strategy must be one of"): redact("test", entities=[], strategy="invalid")