the integration half of #1012: one receipt, three borrowed consumers

[AdaWorldAPI]

PROBE-TOKEN-SEAM-1 — 41 gates, 18 disable-runs each verified red-then-green — asks what E-TOKEN-BPE-CAN-FIT-NOT-YET-BUY-1 (#1012) explicitly did not: can ONE versioned BPE tokenization of a source span serve several consumers at once?

Measured yes. One source tokenization per span — Tantivy, DeepNSM-v2 and a forward-prediction input surface each added zero. Query analysis is counted on a separate counter, because a query is different bytes and folding it into one number would make the claim a lie.

Where the code lives — CHANGED 2026-08-24, read this before looking for it.
It is now AdaWorldAPI/tesseract-rs, crates/tesseract-paperless,
feature-gated (default = the S-2 gate + doc.v1 → DocIr; ocr = the
in-process recognizer; token = the lexical seam), pushed and CI-wired on
all three tiers. It was written in AdaWorldAPI/paperless-rs, whose push is
denied at the GitHub-App level (403 through the session proxy AND
proxy-bypassed; the in-environment GH_TOKEN is a 14-character placeholder,
so the usual "a 403 here is the proxy" escape does not apply), so it had no
durable home there. tesseract-rs is the one source of truth; paperless-rs
is a dead copy. The patch under .claude/harvest/paperless-token-plateau/
is now an ARCHIVE — forensics, not a way to obtain the code — and its README
says so.

This PR is the board record for that work: the epiphany, the arc entry, the state table, and the archived patch.

The identity comes from ogar-doc-ir — the receipt mints nothing

The first cut invented source_id: u32 / span_id: u32. That was a second population wearing the document layer's job; ogar_doc_ir::DocIr already answers all three questions a receipt asks, and the seam was re-cut to read them:

the receipt needs	ogar-doc-ir supplies
WHICH document	DocIr::content_sha256, interned once per document
WHICH span	a Region, at (DocPage::number, Region::reading_order)
the span's text	Region::text

Two consequences worth the PR:

• content_sha256 is a PER-ACQUISITION dedup key, not a cross-retina identity — the crate's own docs correct its plan's first sketch on exactly this point. For a tokenization receipt that is the right reading: you tokenize bytes, so different bytes are a different tokenization. Cross-retina convergence is a facts question (converges_on_facts) and is not this seam's business.
• "The OCR boundary supplies no byte offsets" is RETIRED as a gap. It supplies no page-wide offset and does not need to: a region owns its text, so an offset is region-local, and ogar-from-docv1::region_text is where the leading_space-aware join already lives. What remains is far smaller — a sub-region span needs a non-zero byte_from, which the receipt already carries and no producer emits.

The seam is source-agnostic for free: docir.rs has no line that knows which retina produced the IR.

The two facts that make it adoptable, and neither was designed for it

1. DeepNSM-v2's library is already the seam. parse_to_spo(&[Tagged]) consumes (WordId, Pos) pairs and touches no string; the split_whitespace/normalise logic lives ONLY in its two examples. The crate needed no change — and in the new home that shows up structurally: deepnsm-v2 is a dependency of the PROBES and not of src/.
2. Tantivy structurally cannot own offsets. Its indexer reads Token::text and Token::position, uses position_length transiently, and reads offset_from/offset_to nowhere outside its own tests. An index cannot become the ABI even by accident.

The number that bounds prior work

	KJV scene	Alice
bytes / spans / tokens	1 125 / 8 / 352	75 513 / 300 / 36 994
compression	3.20×	2.04×
trained table	180 of 255 (corpus-bound)	255 of 255 (cap-bound)
resident lane	864 B (77 % of source)	55 400 B (73 %)
receipt share of resident	~52 %	~30 %
tokens per lexical unit p50/max	1 / 7	2 / 15

The 8-bit id table is FULL at 75 KB of ordinary English (and still full on the whole 170 KB file), while on the KJV fixture the corpus rather than the cap set it at 180 — which is what #1016's own record of that fixture says. The two corpora sit on opposite sides of that line. #1012's 3.35× is a 1 KB figure and must not be read as corpus-independent.

(An earlier revision of this body quoted "247 of 255" as the vocabulary. That was the count of ids appearing in the lane, not the table size. Corrected here and in the board entry.)

The follow-up probe named there has since RUN, and it inverts the headline: at the 256:256 rail (one token per (u8:u8), two separate bytes, never a widened u16) alice-full compresses 6.13× against the flat-u8 1.95×, uses 7 675 of 65 536 ids — corpus-bound with 88.3 % of the tile free, where the u8 table was CAP-bound — and its resident particles are 35.2 % smaller. So "saturates at 75 KB" was the cap's property, not the corpus's.

Gaps, named rather than worked around

1. No shipped token continuation mechanism. rail_geometry::RailCarving::AxisSlab { reg, cont } caps at RAIL_MAX_DEPTH = 24 — below the measured p50 of 4 particles. Two framings are lawful and the trade is exact: particle_count alone bounds the run and a PAD scan inside it is already exact because PAD is reserved (cost: one vocabulary slot, which at a cap-bound table is not free); or token_count costs 4 bytes and frees the slot.
2. ValueTenant has no token variant (16 discriminants, none for text).
3. No callable part-of-speech surface exists — deepnsm_v2::lexicon was deliberately deleted, and the insight_coca_read grounding cited for that deletion is itself an example binary in a crate carrying serde/serde_yml/tokio/ndarray, outside a lean consumer's dependency barrier. Recorded rather than re-litigated.
4. The semantic half never ran — cam96_codebook.bin / cam96_codes.bin are absent, so palette256² distance is unexercised.
5. The probe builds its DocIr from text, not from a retina. Half-closed — see below.

Update 2026-08-24 — the move, and what it changed on purpose

Four commits landed before the code moved (the seam probe; the ogar-doc-ir re-cut; the two measurement probes; the intake wiring). The archived patch carries all four. In the new home three things are deliberately different, and each is a fix rather than a port:

The whole [patch] section is gone. tesseract-rs path-deps its siblings, so the escaping-relative-path trap does not arise — and with it goes the branch-vs-rev problem: cargo does not unify a branch and a rev source even at the identical commit (measured: two ogar-doc-ir entries in the lock, same #719471db suffix), and two SourceIds mean two incompatible DocIr types in one binary, which defeats a source-agnostic IR entirely.

ingest_html became ingest_doc_ir(bytes, index, build). The old form called spider_doc_ir directly, which bound intake to ONE crawler and inherited its build — and that build is broken (E0063: missing field 'confidence', a field ogar-doc-ir added after spider's only commit; spider floats on branch = "main", so it broke the moment the field landed and nothing builds the pair to notice). The new form takes the producer as a closure: any producer that can build a DocIr is admitted and the crate depends on none of them. That is what a source-agnostic IR is FOR, and hard-coding one crawler had quietly spent the property. The closure — rather than a ready-made DocIr — is what keeps S-2 true for a producer the crate has never heard of, because on a duplicate it is never called.

One assertion became a runtime guard. The old code merely tested that the producer's hash matched the gate's. IntakeError::IdentityMismatch now refuses the mismatch, because a producer keying its IR by a normalised or re-encoded form of its input would leave the gate and the subtree silently addressing two different documents.

Both new guards are disable-verified, 12 of 13 passing under each, so each is pinned by the test that names it rather than by collateral damage:

assertion	disable	observed
a producer keyed by other bytes is refused	delete the content_sha256 != hash check	that test, and only that test, FAILED
a duplicate never reaches a third-party producer	move build() ahead of the short-circuit	that test FAILED, on the closure's own panic

The evidence reproduced in the new home, checked rather than trusted: probe_token_seam reports ALL 41 GATES GREEN with numbers identical to the original run, and both measurement probes reproduce. 13 lib tests; all three tiers clippy-clean at -D warnings; fmt clean; tesseract-core 27/27 unaffected.

Gap 5 is half-closed. Intake now runs the gate in front of real producers and hands back one DocIr either way. What is still open is the comparison that would actually test source-agnosticism end to end — the SAME content through two different retinas — and it is blocked on the upstream one-liner in spider, which this session cannot push.

Worth carrying past this PR: a floating branch dep with no CI that exercises it is drift with no detector. ogar-doc-ir's closed RegionKind vocabulary and version marker guard the DATA shape at load time; neither can see a struct field added at compile time. The new crate is a workspace MEMBER with optional deps for exactly this reason — an excluded crate is never compiled by CI, and an uncompiled crate rots invisibly.

Polars

Refuted, and the honest form is weaker than the question invited: zero occurrences across nine checkouts; every DataFrame mention is prose; neither the intake tree nor tesseract-rs declares arrow/datafusion/lance/lancedb at all. There was nothing to remove.

Method notes

An independent vacuity audit of the finished probe found five holes, all real — a gate asserting byte counts but never span counts (the CRLF bug that collapsed 300 spans into 1 would have re-passed it), a threshold true by construction, an assertion about a type signature rather than behaviour, an unconditional prefix check, and an unexercised ASCII-vs-Unicode whitespace divergence. All fixed.

Three disable-runs were themselves wrong first. The sharpest is from the re-cut: T-DOCIR-KEY compared each receipt against the same spans() call it was validating — an implementation checked against itself — and stayed green when spans() was changed to renumber by position. It now walks the DocIr independently, and the fixture's reading_order is 2i+1 rather than the positional index so the two are distinguishable at all. A fourth disable (flattening table cells) found nothing because neither text corpus contains a table or a figure — a fixture-coverage gap, closed with a purpose-built three-region page.

A knob that does not bind is not a disable; a fixture's SHAPE is part of a test's coverage; and an implementation compared against itself is not a check.

Board hygiene

EPIPHANIES (E-ONE-RECEIPT-MANY-BORROWED-CONSUMERS-1, with its in-entry self-correction), PR_ARC_INVENTORY, LATEST_STATE, AGENT_LOG, and the archived patch + its superseded notice.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ArVbbq3DsToBM7r79zGeEf

[chatgpt-codex-connector]

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_063382cf-9de5-49ff-a8e2-3e60b017c3b2)

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: bcc29eb4-d501-4a06-8f0d-adbfbfa97fa7

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cursor]

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_1733c764-990a-4e53-a545-838a662c899e)