ship/census spanscan - #5704
Merged
Merged
Conversation
matthewevans
commented
Jul 13, 2026
Member
- fix(scripts): teach the census scanner Rust raw strings
- perf(scripts): scan the census seam span-wise, not character-wise
`strip_noncode` -- the scanner seam shared by the zone-authority census and
the Draw-replacement census (via `iter_production_lines`) -- had no branch for
raw string literals. Their contents leaked into `code` as if they were source,
which is the ceiling on every full-tree scan: `crates/draft-wasm` could not be
scanned AT ALL (`CensusError: brace tracking desynced leaving a #[cfg(test)]
mod`, from the brace-bearing JSON fixture at suggest.rs:437).
Rewrite it as a cross-line lexer that handles:
* `r"..."`, `r#"..."#` at any `#` depth (only `"` + the SAME count closes)
* the byte/C-string forms `br"`, `br#"`, `cr"`, `cr#"`, plus `b"` / `c"`
* multi-line raw strings, WITHOUT dropping the remainder after the close
* nested block comments (`/* a /* b */ still a comment */`), which Rust
allows and a bool `in_block` cannot represent
`ScanState` replaces the `in_block: bool`: both constructs that span a line
boundary need a COUNT, not a flag.
The `#[cfg(test)]` region marker now keys on `code` rather than `raw`. This one
is belt-and-braces, and is recorded as such: no witness could be built for it,
because `pending_cfg_test` is already cleared by any line with non-empty code
and a raw-string terminator always leaves at least a `;` behind. That is a
coincidence of the terminator's punctuation, not a declared property.
Both frozen baselines are byte-identical (zone 62 rows / 82 hits, draw
producers 7 rows) -- the covered crates' 103 raw-string sites are all inert, so
any movement would have meant the change was wrong.
Tested at the seam (`zone_authority_census_tests.py`, 24 cases, stdlib
unittest): every raw-string form above, plus the historical failure shapes -- a
raw string whose `//`/`/*` must not open a comment, a multi-line raw string
whose remainder must survive, and a quoted `#[cfg(test)]` that must not toggle
region state. 14 of the 24 are red against the pre-fix scanner.
The raw-string lexer was correct but walked the line one character at a time,
firing a compiled-regex match per character. At the volume this seam runs at
(~7MB of Rust per census, 1.5M lines tree-wide) that is not a micro-cost: it
made the scanner the gate's bottleneck.
Three changes, no semantic ones:
* CANDIDATE jumps to the next position that could OPEN a comment or literal
(`/`, `"`, `'`, or a b/c/r prefix with a quote behind it) and the code
between two candidates is taken in ONE slice. Every alternative starts with
a character class so the regex engine can prefilter at C speed -- the token
boundary is checked in Python afterwards, deliberately NOT as a lookbehind,
which would defeat the prefilter.
* A fast path for the 4-in-5 lines that hold no comment and no literal at
all: return the line untouched, allocating no list, no join, no ScanState.
* The hot loop hoists the bound pattern methods and `len`, and dispatches on
one character read instead of two `startswith` calls.
Measured on the 5 largest engine/src files (6.75 MB), CPU time, best-of-5 --
wall clock is meaningless on a box running concurrent cargo builds:
pre-fix scanner (b9685cc) 6.24 MB/s
raw-string fix, per-char 3.25 MB/s
this commit 33.12 MB/s 10.2x / 5.3x
The zone gate's own `collect()` drops from 8.67s to 2.91s CPU -- 3.0x faster
than before this unit began, so the raw-string fix now costs less than nothing.
Equivalence is the invariant, and it is proven, not asserted: run against the
previous (verified byte-identical) lexer over EVERY .rs file in crates/ --
1705 files, 1,495,438 lines -- both `code` and `ScanState` agree on every
single line, 0 divergences. The 24-case suite is green and both frozen
baselines remain byte-identical (zone 62 rows / 82 hits, draw producers 7).
matthewevans
enabled auto-merge
July 13, 2026 00:04
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
matthewevans
added a commit
to lgray/phase
that referenced
this pull request
Jul 13, 2026
* fix(census): lex a Rust lifetime as a lifetime, not as a char literal
strip_noncode's char-literal alternative was permissive -- `'(?:\\.|[^'\\])*'`
-- so it could not tell a char literal from a LIFETIME (`&'a str`, `Foo<'_>`) or
a loop LABEL (`'outer:`), neither of which any second quote ever closes. Given an
odd number of ticks on a line, the tick pairs with the opening quote of the next
char literal and everything between them is eaten as literal content:
fn drain(&'a mut self) { self.hand.push_back(c); assert!(c != 'x'); }
-> fn drain(&x'); } <- the hit AND the brace, gone
That is the raw-string ceiling again (phase-rs#34), with the same two failure modes: a
SWALLOWED HIT, which reads like migration progress rather than a mis-scan, and a
LEAKED BRACE, which desyncs brace tracking for the rest of the file. The tree has
a live instance of the second, in census scope:
engine/src/parser/oracle_replacement.rs:9560
char::<_, OracleError<'_>>('{'), -> char::<_, OracleError<{'),
which invents a `{` and drifts depth for ~9,600 lines. It has been harmless only
because the drift is a constant offset that no #[cfg(test)] skip region straddles
-- luck, of exactly the kind phase-rs#34 removed.
Encode Rust's own rule instead: a char literal is ONE char or ONE escape ('x',
'\n', '\x41', '\u{1F600}') followed by the closing quote. A quote that does not
close that way opens a lifetime, and its tick is emitted as the ordinary code it
is. The CANDIDATE prefilter is untouched, so no per-character scanning returns.
Evidence (population stated, computed this session):
* whole-tree sweep, 1,713 .rs files / 1,512,084 lines under crates/: old vs new
diverge on 746 lines. Adjudicated: 1 changes brace balance (the fix above),
0 change census hit count anywhere. The rest is inert text -- an apostrophe
the old lexer ate and the new one leaves alone.
* both frozen baselines stay byte-identical: zone 87 hits / 64 rows, draw
producers 8 hits / 7 rows, unchanged. Both gates exit 0.
* throughput, time.process_time() best-of-5 over the 5 largest engine/src files
(6.78 MB): 35.2 MB/s before, 35.1 MB/s after -- noise, and 7x the 5 MB/s floor.
Five new seam tests, each observed RED against the pre-fix scanner (four assertion
failures and one CensusError from the #[cfg(test)] desync), plus one preservation
guard pinning that every char-literal escape form is still consumed whole.
* ci(census): gate the census scanner's own seam suite, and un-stale the draw-census scope note
Gates (B) and (C) of check-engine-authorities.sh both stand on strip_noncode,
but the 30-case suite that pins that lexer was not run by ANY CI job -- a lexer
regression would not fail the gates, it would silently mis-scope them (a
swallowed hit reads as migration progress, not as a mis-scan). Run the suite in
the same job, ahead of the gates it protects. Cost: 1ms.
Also correct the (C) scope comment: the raw-string blocker it cites was fixed
in phase-rs#5704, so "cannot yet read the whole workspace" is no longer true. The
3-crate scope itself is unchanged -- widening it moves the frozen producer
population and is deliberately left to its own unit.
---------
Co-authored-by: matthewevans <matthewevans@users.noreply.github.com>
This was referenced Jul 13, 2026
matthewevans
added a commit
to minion1227/phase
that referenced
this pull request
Jul 13, 2026
…rkspace (phase-rs#5712) Gate (C) in check-engine-authorities.sh froze the `ReplacementEvent::Draw` producer surface by scanning 3 named crates (engine, engine-wasm, mtgish-import) of a 13-crate workspace. That list was never a decision — it was a workaround for a scanner ceiling: `strip_noncode` had no branch for Rust raw strings, so a workspace-wide scan died on `crates/draft-wasm/src/suggest.rs:437: brace tracking desynced`. phase-rs#5704 taught the scanner raw strings, so the scope can now be what it always claimed to be. SCOPES is now globbed (`crates/*/src`), not enumerated. A hand-written tuple is a list that goes stale the moment crate phase-rs#14 lands — which is exactly how this census went wrong twice before (v1 froze 6 producers while a 7th was live in mtgish-import). The glob cannot go stale, and because the producer baseline is an exact-match gate, a Draw producer minted in a brand-new crate fails the build on the day it arrives. Population: 3 -> 13 crates, 469 -> 701 .rs files scanned (+232). Producers: 7 rows -> 7 rows. ZERO new producers. `ReplacementEvent::Draw` occurs in exactly two crates (engine, mtgish-import), so no producer was missing a CR 121.2 scope decision and no row's scope moved. The baseline diff is header prose only. The zero is not vacuous. phase-ai builds `ReplacementDefinition`s with the very same constructor and struct-literal idioms this census matches (AddCounter, Moved, ChangeZone) — just never with `Draw`. And a synthetic `Draw` producer planted in phase-ai turns the widened gate RED (`ADDED crates/phase-ai/src/cast_facts.rs scratch_t59_synthetic_draw_producer constructor 1`) while the OLD 3-crate scope reports `PASS (7 rows)` — the widening is what catches it. Boundary, named rather than implied: the two Rust trees Cargo excludes from the workspace (`client/src-tauri`, `lobby-worker/broker-wasm`; 4 .rs files) are not scanned. Neither mentions `ReplacementEvent`. `client/src-tauri` also grows a gitignored `target/` on any local Tauri build, which an rglob would descend into — making the scanned population depend on whether a developer had run one. CR 121.2 (grep-verified, docs/MagicCompRules.txt:1144): cards may only be drawn one at a time. CR 121.2a (:1146): an instruction to draw multiple cards can be modified by replacement effects that refer to the number of cards drawn. Co-authored-by: matthewevans <matthewevans@users.noreply.github.com>
matthewevans
added a commit
to minion1227/phase
that referenced
this pull request
Jul 13, 2026
…n strip_noncode (phase-rs#5715) The census lexer closed a non-raw string only if its quote closed on the same line. Rust's STRING_CONTINUE says otherwise: a `\` immediately before the newline escapes it and the literal RUNS ON -- which is how every wrapped `assert!` message in the tree is written. Scanned as code, the continuation body was the raw-string ceiling (phase-rs#5704) and the lifetime ceiling (phase-rs#5705) a third time, and it failed in all three of their directions at once: a `//` in the body opened a phantom comment (`Assault // Battery` is real message text in this tree), a `{` LEAKED into the brace counter, and an `add_to_zone(` in a message was a FALSE HIT. Population, measured (instrument: an independent Rust lexer over the census scopes, 434 scanned files / 919,557 lines): 1,114 continued-string lines in the files the census scans, 872 more in the name-excluded ones. On production lines the old lexer emitted 204 string-body lines as code and leaked 160 braces out of them. The leak was live-by-luck: they balanced, so no skip region moved -- an unbalanced one desyncs brace tracking and either raises CensusError or silently swallows the production code after it. ScanState now carries the third multi-line construct alongside block-comment depth and raw-string `#` count. The three are mutually exclusive by construction: a `\` is inert inside a raw string (no escapes at all), and either string inside a block comment is comment text. Old-vs-new whole-scope sweep: 204 code-text divergences, ALL of them string body that is no longer scanned as code; 0 lines added to or removed from the production stream; 0 hit-set changes; 0 CensusError either way. Both baselines (zone, draw-replacement) byte-identical, `--write` idempotent, all three gates in check-engine-authorities.sh green. Throughput 18.8 MB/s CPU (best-of-5, process_time, 5 largest engine/src files), up from 17.9: a continuation line now short-circuits instead of being scanned character by character. Tests: 11 red-first cases in the seam suite (8 failures + 1 CensusError against the old lexer) covering false hit, leaked brace, phantom line comment, phantom block comment, 3-line continuation, cfg(test) region toggle and desync, the b/c prefixed forms, and the minimal pairs an escaped backslash and a raw string make. Co-authored-by: matthewevans <matthewevans@users.noreply.github.com>
andriypolanski
pushed a commit
to andriypolanski/phase
that referenced
this pull request
Jul 13, 2026
…aw text (phase-rs#76) (phase-rs#5718) Family (D) of the parser-combinator gate finds an `alt((...))` block's end by counting parentheses over RAW text. That count is a lex, and a lex that reads literals and comments as code is the ceiling class that has now bitten this repo's census scanner three times (raw strings phase-rs#5704, lifetimes phase-rs#5705, backslash-continued strings phase-rs#5715). Here it has teeth in BOTH directions, because the gate it feeds is commit-blocking: FALSE HIT an `alt((` written inside a COMMENT opens a phantom block, which swallows the ordinary tag() bindings below it and blocks a good commit over an `alt` that does not exist. FALSE MISS a comment carrying `))` exhausts the counter's depth headroom and truncates a REAL block: the arms are never collected and a genuine cross-product alt ships past the gate built to stop it. This is string-matching parsing landing on main with a green gate. Both are witnessed and committed as tests. In-tree today the bug is live-by-luck: 9 of 2,853 alt blocks in crates/engine/src/parser have their end line computed wrong (all of them phantom blocks opened by an `alt((` inside a comment), but zero FLAG DECISIONS change. Equivalence sweep old-vs-new over every block, with every line treated as added: 0 decision changes, and the block count falls 2,853 -> 2,844, exactly the 9 phantoms and nothing else. Positive control on the sweep harness: it reports a decision change on both witnesses, so the zero is a finding rather than a dead probe. DEPENDENCY DIRECTION, deliberately narrow: family (D) now imports the census module's lexer (scripts/zone_authority_census.py). Single authority for CODE-STREAM lexing -- explicitly NOT for literal matching. The detector reads STRUCTURE (where does `alt((` open, where do its parens balance?) from the stripped code stream, and CONTENT (which literals are the arms? is there an allow-noncombinator annotation?) from the raw text. Two views, one grammar. The same commit documents why the gate's OTHER families must NOT follow. (A), (B), (E) and (F) match ON the string literal (`.contains("`, `"lit" =>`, `== "long"`, `name: "..."`); routing them through strip_noncode would delete the very quote they key on and BLIND the gate -- it would pass string-matching parser code forever while reporting green. Measured during phase-rs#76: 194 real match arms go unmatched that way. Their exposure to this class is the benign mirror image, a false hit that is loud and already escape-hatched (`// allow-noncombinator:`); phase-rs#76 measured 10 such lines in the whole parser scope, all doc comments describing the forbidden pattern. A comment block at the grep definitions pins this where a future "hardener" would edit. The detector's seam suite runs from the gate (section D0), ahead of the scan it protects -- the same shape as section (B0) of check-engine-authorities.sh, and for the same reason: a lexing regression here does not FAIL this gate, it silently mis-scopes it. Watched red (a deliberately broken suite fails the gate) and green. Zone and draw baselines are untouched and byte-identical; this change is gate-internal. Audit outcome for the other scanners in phase-rs#76: scripts/audit-parser.py is NOT A MEMBER (it never opens a .rs file -- its only input is card-data.json), and scripts/gen-test-fixture.py is benign-directional (over-harvest is fixture bloat; a missed card is loud at test runtime). Neither is changed. Co-authored-by: matthewevans <matthewevans@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.