Loop trace, small-window support, and ten harness fixes it found - #789
Merged
Conversation
added 11 commits
August 14, 2026 23:16
--trace <path> writes a JSONL record per loop decision: turns, tool calls, token usage, the context manager's verdict, and every harness intervention attributed to the guard that sent it. scripts/loop-trace.py renders it. It taps the LoopEvent stream at the pump in integration.rs, the one point every event passes through on its way to every consumer, so there is no second set of call sites to keep in step with the first. describe() is exhaustive, so a new LoopEvent variant won't compile until it says how it traces. Interventions are named from the existing tag registry, so a guard added later is traced without anyone editing this file. Three bugs found by running it against a local model: A force-ended turn ended the whole run. The ExitWithSummary tier sets force_turn_end, which broke out of the inner loop — but the inner loop is the turn loop, so control fell to the finalization poll and the run stopped. A model over the threshold got one turn: its tool calls ran, results were appended, and the run ended before it ever saw them, silently. vpma.22 in #779 introduced this; the arm did nothing at all before that. Now it continues the run when the fold actually made room, and when it didn't, stops with a notice naming prompt_tokens and ctx_max. glm-5.3 matched nothing in the context-window table (glm-5.2 is listed at 1M) and fell to the 128k default, so a live session showed 226.9k/128.0k and "compaction soon" on a model with eight times that room. Every context tier divides by that number. Family prefixes now cover point releases, qwen3/qwen4 no longer inherit the original qwen's 32k, and a model neither lookup knows warns once instead of being guessed in silence. turns= on the gates line read 0 for any run that force-ends its turns: record_turn sat past the break. Moved to where the turn happens. The fold and force-summary log lines now carry prompt_tokens and ctx_max, which took solving a division by hand to recover.
The verifier declines a build/test command whose exit status was piped or sequenced away — the zero belongs to tail, not pytest — and that is right. But it returned without recording why, so status() fell to the "nothing ran" branch and the model was told "you didn't run the tests or build to check it" after running pytest four times. Measured against a local model: it did not remove the pipe, because the sentence was visibly false. It re-ran the same shape twice more, then added `; echo "exit=$?"`, which reports echo's status. The claim gate fired on the consequence and the final answer asserted "exit status 0" — a number it never had. Three of fifteen turns, ending Unverified with a green suite. The verdict is unchanged: a masked pass still is not a green. Only the message changes — it now quotes the command, says the status belongs to the last stage of the pipe, and says what to run instead.
The summary counted assistant message records, which the trace stopped writing when the pre-stream placeholder was dropped — so it read 0 turns for a run that took 15. turn_end is one per completed turn and matches the gates line.
A local model was observed thinking for thirteen minutes across 2892 tokens with nothing recorded between turn_start and the next event — which reads like a deadlock in a file whose stated purpose is explaining a run that hung. MessageUpdate now emits one throttled record per 10s saying what the turn is doing and how much it has produced. One record per delta would be the opposite failure, so the throttle is asserted both ways.
context_window (GH #772) is now settable per provider. It was top-level only, so with more than one provider configured — the common case — there was no way to correct one model's window without corrupting the others'. Correcting a local model's window this session needed a whole separate config directory for exactly that reason. Precedence is providers.<name>.context_window, then the top-level key, then the model table, then 128k. The loop's window override and the session's gauge now resolve from the same call, at the one point where provider and model are both final. It stays None when nothing is configured, so the loop keeps re-resolving from the table per run and a mid-session /model switch still tracks the new model. Separately: claim_gate::segment_kind took the first token of a segment as the command, so `python3 -m pytest -v` classified as `python3` and returned None. The verifier matches on any token, found `pytest`, and recorded the run green — so the model was told "no build/test command of the matching kind ran this run" one turn after running one correctly, having just been corrected by the verifier for running it wrong. Measured live. segment_kind now steps past an interpreter or runner prefix: python -m <module>, npx/bunx, poetry/uv/pipenv/rye/hatch run. `python script.py` and `python -c '…'` are deliberately left alone — the interpreter is running a script, not a known tool. Neither recogniser knew `unittest`; both do now. A test asserts the two agree on the commands where they overlap.
The gauge divided total_estimated_tokens — a chars/4 heuristic plus per-tool-call overhead over the whole persisted transcript, tool results at full length — by the model's advertised window. The context manager compares the provider's prompt_tokens for the REQUEST, whose oversized results have been snipped and whose old turns are a summary, against effective_ctx_max. Reported live: 226.9k/128.0k, 100%, 'compaction soon', one compaction, on a run whose requests were never close. Session now keeps last_prompt_tokens, set from usage.input_tokens alone — the same field decide_after_usage divides by. Not the sum of the three input figures: that would be closer to the true prompt on Anthropic and double-count on DeepSeek, and either way put the gauge back to describing a fold that does not happen. The estimate stands in until the first response. Both the panel and the status bar read it.
Told to drop the `|` or `;`, one model produced a clean `pytest -v` and another produced `pytest -v; echo "EXIT=$?"` — which obeys the letter, tries to surface the status, and masks anyway. The nudge is now positional (the build/test command last, nothing after it) and names the three idioms that have actually shown up. Also records the long-horizon run in docs/harness-review-2026-08.md, including a cascade it exposed: a masked-but-green test run means verified_green never latches, so the progress monitor — which counts a green as one of its three progress events — fired the stall nudge twice at a model that had passed all 22 tests, the second time at 618.0s of a 618.1s run. Filed as hwk9.4 rather than patched: my first fix blamed edits_since_verify, which the stall nudge does not read and which an existing test deliberately pins the other way.
added 3 commits
August 15, 2026 01:03
…stream
Tracing what the front end receives, not just what the loop decided, showed
the TUI rendering every harness intervention's body twice. The notice
carries "harness intervention: {summary}\n{body}" because headless sees only
it — --print renders SystemNotice and ignores UserMessage entirely — but the
TUI gets both and renders the body from the message too. The notice now
shows its summary line only in the TUI; the body stays on the message path,
which is the copy dirge-m10x guarantees survives the next stream anchor.
Headless output is unchanged.
With that settled, boundary nudges emit MessageStart/MessageEnd like the
finalization path. They were pushed straight into context with only a
SystemNotice, so stall, budget, prologue, track-work, file-touch,
safe-state, fast-verify and reflection nudges were absent from the message
stream — the tally read nudge_progress_stall=2 while the trace recorded one
intervention. Adding them before the render fix would have put the body on
screen a third time.
Also: the stall checkpoint stands down while the verifier holds a masked
decline. Measured — a run that passed all 22 tests at 345s via
`pytest … | tail -28` was told twice it had made no progress for three
turns, the second time at 618.0s of a 618.1s run. The verify nudge owns that
state and has the actionable message; the stall text offers "getting a green
check" as the way out, which is what had just happened and could not be
counted. The prologue checkpoint is not suppressed — it fires on a run that
produced nothing, where a masked verification is not the explanation.
dirge's opening request is large before the model has done anything: measured, 16,172 prompt tokens with the 34 built-in tools and 32,621 with MCP servers loaded. The second is larger than a 32k window in its entirety, so such a run could not take a single turn — and the only symptom was the context manager force-ending every one. On a window at or below 48k, each tool's description is trimmed to its first sentence and each parameter's to a short clause. Names, types, enums and required-ness are untouched: the model keeps everything it needs to form a well-formed call and loses the prose about when to prefer one tool over another. No tool is dropped — a model that cannot see a tool cannot ask for it, and that failure is silent and looks like incapability. Measured on the same task and model at a 32k window: 16,202 -> 12,249 prompt tokens, context peak 51% -> 39.5%, and the model still reached for list_symbols unprompted and answered correctly. config `compact_tool_schemas`: auto (default) / on / off. The decision is made at loop_tool_to_rig_definition, the one point every tool becomes a provider schema, and sized against the same window the session gauge and compaction use — passed in rather than re-derived.
Measured on deepseek: the model ran a clean `python3 -m pytest -q` and reported "Confirmed with a real exit status: 22 passed in 0.01s, exit 0." The claim gate fired, because "exit 0" sat in the build/lint list and no build had run. The verify nudge is what asks for the exit status when it declines a masked run — so one guard requested the number and another penalised the answer. "exit 0" / "exit code 0" / "exit status 0" are now satisfied by any observed verification. Kind-matching stays where it earns its keep: a build still cannot support "N passed" and a test run still cannot support "clippy clean", both pinned by a test. Found by running the same task on qwen, deepseek and glm to check the earlier fixes weren't shaped to one model. All three passed 22/22, and all three piped their test output through `tail` — masking is what models do, not a small-model quirk.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
--trace <path>writes a JSONL record per loop decision — turns, tool calls,token usage, the context manager's verdict, what the front end receives, and
every harness intervention attributed to the guard that sent it.
scripts/loop-trace.pyrenders it;docs/loop-trace.mdexplains how to readone and
docs/harness-review-2026-08.mdis the full write-up with evidence.It taps the
LoopEventstream at the pump inintegration.rs— the one pointevery event passes through — so there is no second set of call sites to keep in
step.
describe()is exhaustive, so a new variant won't compile until it sayshow it traces. Interventions are named from the existing tag registry.
Everything below was found by running dirge end to end: first against a 27B
Qwen on llama.cpp, then the same task on deepseek-v4-flash and glm-5.3 to check
the fixes weren't shaped to one model. All three passed the 22-test task, and
all three piped their test output through
tail— masking is what modelsdo, not a small-model quirk.
The pattern
Every guard's reasoning was right. Several described themselves wrongly, and in
each measured case the model did not comply — it worked around the guard, once
into a fabricated claim. Same shape as
docs/verification-discipline.md's"a guard that cannot explain its scope gets routed around, not satisfied".
Loop control
A force-ended turn ended the whole run, silently. The
ExitWithSummarytier breaks the inner loop — which is the turn loop, so control fell to
finalization and stopped. A model over the threshold got one turn: its tool
calls ran, results were appended, run over before it saw them. It only appeared
to work because a gate happened to fire and restart the outer loop. vpma.22 in
#779 introduced this. Now it continues when the fold made room and stops with a
notice naming
prompt_tokensandctx_maxwhen it didn't.turns=read 0 for any run that force-ends — the denominator every othercount on the gates line is read against.
Context accounting
Per-provider
context_window(GH #772). Top-level only meant no way tocorrect one model's window without corrupting the others'. The loop's override
and the session's gauge now resolve from the same call. Verified live.
An unknown model silently got 128k.
glm-5.3matched nothing (glm-5.2islisted at 1M); a local model is named by its file path, so
…/Qwen3.8-27B-Q8_0.ggufmatchedqwen→ 32000, smaller than dirge's ownprompt. Family prefixes now cover point releases; an unknown model warns once.
Verified live — glm-5.3 now resolves to a 250k effective window.
The context gauge measured the wrong thing.
226.9k/128.0k, 100%, compaction soonwith one compaction: it divided the persisted transcript'sestimate by the advertised window, while compaction compares the request's
prompt_tokensagainsteffective_ctx_max. Now reads the two numbersdecide_after_usageactually compares.Breadcrumb tool schemas below a 48k window. dirge's opening request is
16,172 prompt tokens with the built-in tools and 32,621 with MCP servers — the
latter larger than a 32k window entirely, so such a run could not take a single
turn. Descriptions are trimmed to their first sentence; names, types, enums and
required-ness are untouched, and no tool is dropped. Measured at 32k: 16,202 →
12,249 tokens, peak 51% → 39.5%, and the model still selected
list_symbolsunprompted.
compact_tool_schemas: auto / on / off.Guards that misdescribed themselves
"You didn't run the tests", after four passing runs. The verifier declined
pytest … | tail -12correctly — the exit status istail's — but returnedwithout recording why. The model re-ran the same shape twice more, then added
; echo "exit=$?", and its final answer asserted "exit status 0", a number itnever had. A/B on the same task: old message →
Unverifiedwith a green suite;new message → corrected on the first try,
VerifiedGreen. Confirmed on allthree models.
claim_gateand the verifier disagreed aboutpython3 -m pytest.segment_kindtook the first token, gotpython3, returnedNone; theverifier matches any token and recorded green. So a model that had just been
corrected, complied exactly, and reported truthfully was told its report was
unsupported. Interpreter and runner prefixes are now peeled; neither recogniser
knew
unittest. A test pins that the two agree where they overlap."exit 0" was a build claim. deepseek ran a clean
pytest -qand reportedthe exit status the verify nudge had asked for — and the claim gate demanded a
build. Now kind-agnostic, satisfied by any verification, while a build still
cannot support "N passed".
The stall checkpoint fired at a model that had passed every test. A masked
green means
verified_greennever latches, and the progress monitor counts agreen as one of its three progress events. It now stands down while a masked
decline is outstanding — the verify nudge owns that state. (My first fix here
was wrong: I blamed
edits_since_verify, which the stall nudge doesn't readand which an existing test pins the other way. Reverted and recorded.)
What the TUI renders
Tracing the front-end stream showed every intervention rendering its body
twice: the notice carries summary+body because headless sees only it, while
the TUI gets both and renders the body from the message too. The notice now
shows its summary line in the TUI; the body stays on the message path, which is
the copy
dirge-m10xguarantees survives the next stream anchor. Headlessunchanged.
That made the last fix safe: boundary nudges now emit
MessageStart/MessageEndlike the finalization path. They were pushedstraight into context with only a notice, so stall, budget, prologue,
track-work, file-touch, safe-state, fast-verify and reflection nudges were
absent from the message stream — the tally read
nudge_progress_stall=2whilethe trace recorded one intervention. Verified live:
[stall]now appears as anattributed intervention.
Filed, not fixed
dirge-hwk9.7: the stall checkpoint tends to fire as a successful runconcludes — qwen at 618.0s of 618.1s, deepseek at 55.3s of 55.4s. Behaving to
spec, but spent on a model that is finishing. Needs a decision.
5580 tests pass; fmt and all four clippy configs clean.