ship/census lifetimes - #5705
Merged
Merged
Conversation
matthewevans
commented
Jul 13, 2026
Member
- fix(census): lex a Rust lifetime as a lifetime, not as a char literal
- ci(census): gate the census scanner's own seam suite, and un-stale the draw-census scope note
strip_noncode's char-literal alternative was permissive -- `'(?:\\.|[^'\\])*'`
-- so it could not tell a char literal from a LIFETIME (`&'a str`, `Foo<'_>`) or
a loop LABEL (`'outer:`), neither of which any second quote ever closes. Given an
odd number of ticks on a line, the tick pairs with the opening quote of the next
char literal and everything between them is eaten as literal content:
fn drain(&'a mut self) { self.hand.push_back(c); assert!(c != 'x'); }
-> fn drain(&x'); } <- the hit AND the brace, gone
That is the raw-string ceiling again (#34), with the same two failure modes: a
SWALLOWED HIT, which reads like migration progress rather than a mis-scan, and a
LEAKED BRACE, which desyncs brace tracking for the rest of the file. The tree has
a live instance of the second, in census scope:
engine/src/parser/oracle_replacement.rs:9560
char::<_, OracleError<'_>>('{'), -> char::<_, OracleError<{'),
which invents a `{` and drifts depth for ~9,600 lines. It has been harmless only
because the drift is a constant offset that no #[cfg(test)] skip region straddles
-- luck, of exactly the kind #34 removed.
Encode Rust's own rule instead: a char literal is ONE char or ONE escape ('x',
'\n', '\x41', '\u{1F600}') followed by the closing quote. A quote that does not
close that way opens a lifetime, and its tick is emitted as the ordinary code it
is. The CANDIDATE prefilter is untouched, so no per-character scanning returns.
Evidence (population stated, computed this session):
* whole-tree sweep, 1,713 .rs files / 1,512,084 lines under crates/: old vs new
diverge on 746 lines. Adjudicated: 1 changes brace balance (the fix above),
0 change census hit count anywhere. The rest is inert text -- an apostrophe
the old lexer ate and the new one leaves alone.
* both frozen baselines stay byte-identical: zone 87 hits / 64 rows, draw
producers 8 hits / 7 rows, unchanged. Both gates exit 0.
* throughput, time.process_time() best-of-5 over the 5 largest engine/src files
(6.78 MB): 35.2 MB/s before, 35.1 MB/s after -- noise, and 7x the 5 MB/s floor.
Five new seam tests, each observed RED against the pre-fix scanner (four assertion
failures and one CensusError from the #[cfg(test)] desync), plus one preservation
guard pinning that every char-literal escape form is still consumed whole.
…e draw-census scope note Gates (B) and (C) of check-engine-authorities.sh both stand on strip_noncode, but the 30-case suite that pins that lexer was not run by ANY CI job -- a lexer regression would not fail the gates, it would silently mis-scope them (a swallowed hit reads as migration progress, not as a mis-scan). Run the suite in the same job, ahead of the gates it protects. Cost: 1ms. Also correct the (C) scope comment: the raw-string blocker it cites was fixed in #5704, so "cannot yet read the whole workspace" is no longer true. The 3-crate scope itself is unchanged -- widening it moves the frozen producer population and is deliberately left to its own unit.
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
matthewevans
enabled auto-merge
July 13, 2026 00:52
matthewevans
added a commit
to minion1227/phase
that referenced
this pull request
Jul 13, 2026
…n strip_noncode (phase-rs#5715) The census lexer closed a non-raw string only if its quote closed on the same line. Rust's STRING_CONTINUE says otherwise: a `\` immediately before the newline escapes it and the literal RUNS ON -- which is how every wrapped `assert!` message in the tree is written. Scanned as code, the continuation body was the raw-string ceiling (phase-rs#5704) and the lifetime ceiling (phase-rs#5705) a third time, and it failed in all three of their directions at once: a `//` in the body opened a phantom comment (`Assault // Battery` is real message text in this tree), a `{` LEAKED into the brace counter, and an `add_to_zone(` in a message was a FALSE HIT. Population, measured (instrument: an independent Rust lexer over the census scopes, 434 scanned files / 919,557 lines): 1,114 continued-string lines in the files the census scans, 872 more in the name-excluded ones. On production lines the old lexer emitted 204 string-body lines as code and leaked 160 braces out of them. The leak was live-by-luck: they balanced, so no skip region moved -- an unbalanced one desyncs brace tracking and either raises CensusError or silently swallows the production code after it. ScanState now carries the third multi-line construct alongside block-comment depth and raw-string `#` count. The three are mutually exclusive by construction: a `\` is inert inside a raw string (no escapes at all), and either string inside a block comment is comment text. Old-vs-new whole-scope sweep: 204 code-text divergences, ALL of them string body that is no longer scanned as code; 0 lines added to or removed from the production stream; 0 hit-set changes; 0 CensusError either way. Both baselines (zone, draw-replacement) byte-identical, `--write` idempotent, all three gates in check-engine-authorities.sh green. Throughput 18.8 MB/s CPU (best-of-5, process_time, 5 largest engine/src files), up from 17.9: a continuation line now short-circuits instead of being scanned character by character. Tests: 11 red-first cases in the seam suite (8 failures + 1 CensusError against the old lexer) covering false hit, leaked brace, phantom line comment, phantom block comment, 3-line continuation, cfg(test) region toggle and desync, the b/c prefixed forms, and the minimal pairs an escaped backslash and a raw string make. Co-authored-by: matthewevans <matthewevans@users.noreply.github.com>
andriypolanski
pushed a commit
to andriypolanski/phase
that referenced
this pull request
Jul 13, 2026
…aw text (phase-rs#76) (phase-rs#5718) Family (D) of the parser-combinator gate finds an `alt((...))` block's end by counting parentheses over RAW text. That count is a lex, and a lex that reads literals and comments as code is the ceiling class that has now bitten this repo's census scanner three times (raw strings phase-rs#5704, lifetimes phase-rs#5705, backslash-continued strings phase-rs#5715). Here it has teeth in BOTH directions, because the gate it feeds is commit-blocking: FALSE HIT an `alt((` written inside a COMMENT opens a phantom block, which swallows the ordinary tag() bindings below it and blocks a good commit over an `alt` that does not exist. FALSE MISS a comment carrying `))` exhausts the counter's depth headroom and truncates a REAL block: the arms are never collected and a genuine cross-product alt ships past the gate built to stop it. This is string-matching parsing landing on main with a green gate. Both are witnessed and committed as tests. In-tree today the bug is live-by-luck: 9 of 2,853 alt blocks in crates/engine/src/parser have their end line computed wrong (all of them phantom blocks opened by an `alt((` inside a comment), but zero FLAG DECISIONS change. Equivalence sweep old-vs-new over every block, with every line treated as added: 0 decision changes, and the block count falls 2,853 -> 2,844, exactly the 9 phantoms and nothing else. Positive control on the sweep harness: it reports a decision change on both witnesses, so the zero is a finding rather than a dead probe. DEPENDENCY DIRECTION, deliberately narrow: family (D) now imports the census module's lexer (scripts/zone_authority_census.py). Single authority for CODE-STREAM lexing -- explicitly NOT for literal matching. The detector reads STRUCTURE (where does `alt((` open, where do its parens balance?) from the stripped code stream, and CONTENT (which literals are the arms? is there an allow-noncombinator annotation?) from the raw text. Two views, one grammar. The same commit documents why the gate's OTHER families must NOT follow. (A), (B), (E) and (F) match ON the string literal (`.contains("`, `"lit" =>`, `== "long"`, `name: "..."`); routing them through strip_noncode would delete the very quote they key on and BLIND the gate -- it would pass string-matching parser code forever while reporting green. Measured during phase-rs#76: 194 real match arms go unmatched that way. Their exposure to this class is the benign mirror image, a false hit that is loud and already escape-hatched (`// allow-noncombinator:`); phase-rs#76 measured 10 such lines in the whole parser scope, all doc comments describing the forbidden pattern. A comment block at the grep definitions pins this where a future "hardener" would edit. The detector's seam suite runs from the gate (section D0), ahead of the scan it protects -- the same shape as section (B0) of check-engine-authorities.sh, and for the same reason: a lexing regression here does not FAIL this gate, it silently mis-scopes it. Watched red (a deliberately broken suite fails the gate) and green. Zone and draw baselines are untouched and byte-identical; this change is gate-internal. Audit outcome for the other scanners in phase-rs#76: scripts/audit-parser.py is NOT A MEMBER (it never opens a .rs file -- its only input is card-data.json), and scripts/gen-test-fixture.py is benign-directional (over-harvest is fixture bloat; a missed card is loud at test runtime). Neither is changed. Co-authored-by: matthewevans <matthewevans@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.