Add audit_truncation.py — classify truncation markers by layer - #2
Open
tylorsaling-source wants to merge 1 commit into
Open
Conversation
Companion to audit_ingestion.py for openai/codex#35421. Parses rollout output for both model-facing truncation markers and separates the two surfaces: the exec/unified 'original token count: N' head marker (N varies with real size) from the legacy shell 'N tokens truncated' middle elision. When the 1 MiB capture cap saturates, the middle figure collapses to a constant, so distinct_mid=1 flags capture-cap saturation while a spread flags variable per-call-budget elision. Stdlib only; mirrors the existing audit script's run/arm iteration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #1. Companion to
audit_ingestion.pythat measures the other half of the picture from openai/codex#35421: not how much output was ingested, but which truncation surface fired.It parses rollout output for both model-facing markers and separates them:
Warning: truncated output (original token count: N), emitted by the exec/unified formatter; N tracks real size and varies.N tokens truncated, emitted by the legacy shell middle elision. When the 1 MiB capture cap saturates, the elision input is always exactly 1,048,576 bytes, so N collapses to a constant.So
distinct_mid=1flags capture-cap saturation (stage-1 always saturated), while a spread of middle figures flags variable per-call-budget elision. That column is the difference between your dataset and mine: your batched runs should read head-heavy with highdistinct_mid; a capture-cap-bound workload readsdistinct_mid=1.Stdlib only, mirrors the existing run/arm iteration in
audit_ingestion.py, no changes tobench.py. Runs the same way:python3 audit_truncation.py [runs_dir].Happy to adjust naming or output format to match your conventions — treat this as a starting point for the two-surface view we discussed in #1.
And to be upfront: opening a PR before you've even replied to #1 is presumptuous, I know — I'm hoping to earn the yes by initiative rather than wait for it. No pressure at all; close it freely if it's not a fit.