Harness observability: /trajectory cannot show what the model did

[justrach]

Summary

/trajectory is a DGM fitness ledger (prompt SHA, recipe SHA, parent, score). That job is correct and should stay. It is not a replay of what the model did, and the other local streams do not fill that gap either.

Compared with grok-build's session folder (events.jsonl + chat_history.jsonl + live updates.jsonl) on the same model (grok-4.6, effort high), a real graff session is unreadable as a trajectory. The data is split across three files; two of them lie or lag; /trajectory prints almost none of what they do contain.

Related but distinct: ADR 0019 (transcript shows decisions) and #246 (behavioral schema, closed). This issue is the inspect path — can a human, after or during a run, see the movie.

Evidence (run 28d9d52384b3fc6a, 2026-08-21, grok-4.6 / xAI / 0.0.267)

Session: merge release/v0.0.268 into main, pull/codesign 0.269.

Stream	What it wrote
.graff/trajectories/28d9d52384b3fc6a.jsonl	3 closed turns. Turn 4 (34 API / 98 tools, the live work) never landed.
.graff/traces/28d9d52384b3fc6a.jsonl	~500 api/tool/ws events with name/ms/result_bytes/is_error. No args, no call id, no first_token.
.graff/behavior/28d9d52384b3fc6a.jsonl	1 KB. run_started + turn_started 1–5. No turn_committed, no run_finished, no tool events.
.graff/sessions/merging-0-268-then-pulling-0-269.transcript.jsonl	Full Responses items (args + outputs + assistant text) — closest to grok-build chat_history.jsonl. /trajectory never reads it. Last flushed at turn 3.

Turn 1 DGM row:

tools: "bash,bash,bash,bash,bash,bash,bash!,bash,bash,bash"
ok: true, ms: 95146, model_calls: 10
resp_blocks: 0, resp_text_len: 0, resp_types: ""
cost_microusd: 0

The same turn's recipe_outcome recorded tool_calls: 14. The extra four are meta (todo_write, peer_message, load_tool_schemas, attempt_completion) and never enter ToolSink. The transcript has the assistant narration; the turn node says the model produced nothing.

A no-tool greeting run (c60c75f7…, "hi there") correctly recorded resp_types=text+reasoning, resp_text_len=38. The shape field only works when the history does not end in function_call_output.

Same-day grok-build session of similar length (01a0220a…): 3.7k events.jsonl rows (turn_started / loop_started / phase_changed / first_token / tool_started / tool_completed / MCP start+end), live updates.jsonl, per-call terminal/ logs, signals.json with TTFT/ITL/context %.

Bugs in the recorder (not product taste)

1. finalResponseShape walks backward and stops on function_call_output. src/mainloop_trace.zig #270 was added so a text-only refusal and an empty turn could be told apart (see Agent returns unrelated Chinese refusal during benign coding task #270). On the Responses wire, a tool-using turn's tail is tool results, so every such turn archives resp_blocks=0. The unit tests only cover histories that still end on the model item.

2. DGM turn and session transcript write only on recipe_outcome. An in-flight turn is invisible to /trajectory, /resume, and a crash. Behavior already has turn_started for the live turn; the other two files do not.

3. Behavior lifecycle is incomplete on the interactive path. Schema in src/behavior_trace_types.zig has tool_started / tool_finished / text_delta / turn_committed / run_finished. Default (GRAFF_BEHAVIOR_TRACE unset) is lifecycle-only by design (docs/behavioral-trajectories.md); even that is not emitting turn_committed / run_finished. Rich kinds stay behind GRAFF_BEHAVIOR_TRACE=full — keep that privacy gate.

4. /trajectory ignores fields already on the turn node (tool bag, cache tokens, resp_*, model_calls, tool_errors). It prints turn N ✓ 95s · prompt <sha> · <task>.

Proposed work (ordered)

Do not mix tool events into the DGM JSONL. That file is the population archive; docs/behavioral-trajectories.md forbids merging it with the behavioral stream.

1. Fix finalResponseShape so it shapes the last model-authored run (skip trailing function_call_output). Extend the #270 tests with a Responses history that ends on tool results and still has a prior message + reasoning.
2. Flush a live turn stub when the turn begins (DGM turn with ok omitted / live: true, and a session-transcript checkpoint). Close it on recipe_outcome as today.
3. Teach /trajectory to print what the turn node already has: tools bag (or a count + errors), cache permille, resp_types, and an in-flight marker. Optionally a one-line pointer at the session transcript for that turn.
4. Finish default lifecycle on .graff/behavior/: turn_committed + run_finished. Add first_token + monotonic call_id to the operational trace (ev=tool / ev=api) — counters only, no args/text.
5. Periodic session-transcript flush so /resume and inspect match the wire mid-turn.

ADR 0019 (interpreted one-liners, collapsed infra, compact WORKING) is the display half of the same density problem; land it on its own. Line-REPL click-to-expand is explicitly not in scope (ADR 0019: /debug / TUI fold).

Non-goals

• Default-on GRAFF_BEHAVIOR_TRACE=full (args + assistant text in plaintext).
• Schema-constraining the agentic phase (ADR 0001).
• Copying grok-build's updates.jsonl / rewind snapshots / per-call MCP dumps as a first slice.
• Putting tool payloads into .graff/trajectories/.

Acceptance

• A tool-using turn archives nonzero resp_blocks / resp_types when the model also emitted text or reasoning. The no-tool greeting fixture still works.
• /trajectory during an in-flight turn shows that turn (not "3 turns" while trace is on turn 4).
• /trajectory for a closed turn shows tool count or bag, errors, and cache hit when those fields are present.
• Behavior file for an interactive session that starts and ends cleanly contains run_started, matching turn_started/turn_committed pairs, and run_finished.
• Operational ev=tool rows can be paired by call_id (start optional; finish required).
• Killing the process mid-turn leaves a transcript that includes in-flight function_call items already executed.
• DGM record kinds are unchanged. Old archive readers still parse.

Related

• Agent returns unrelated Chinese refusal during benign coding task #270 — resp_* exists so empty vs text-only can be told apart; this run shows the Responses+tools path still cannot.
• Port schema-harness behavioral trace schema + predict–verify loop into codegraff trajectories #246 — behavioral schema; producer exists, interactive lifecycle does not finish.
• behavior: emit tool_started/tool_finished (with args), action_taken, and text_delta from the tool-dispatch boundary #255 — rich capture opt-in (keep gated).
• ADR 0016 / ADR 0019 — line-REPL density; separate from the inspect path.

[justrach]

Same archive pass: trajectory turn.tools strings truncate at 64 names, so long turns under-count bash. Traces (ev=tool) are complete. If /trajectory grows a “what the model did” view, use traces for tool counts, not the truncated trajectory field.

[justrach]

Two of the five slices landed in #642:

1. ✅ finalResponseShape skips trailing bare function_call_output items, so a Responses history that ends on tool results shapes the last model-authored run instead of archiving resp_blocks=0. New test covers the cut-after-execution case (message + reasoning + function_call under tool results → text+reasoning+tool_use). The no-tool greeting fixture is unchanged.
2. ✅ /trajectory turn rows now print what the node already carried: model calls, tool count, tool errors, cache-hit percent, and resp_types.

Already landed earlier, separate from this PR: item 4's lifecycle half (turn_committed/run_finished on the default behavior stream). Remaining: live turn stubs in the DGM file (item 2), periodic session-transcript flush (item 5), and first_token+call_id pairing on operational traces.

[justrach]

Closed on release/v0.0.279 at 7c49d97 (main fix b24d58c, plus d630588). Acceptance now covered: tool-using turns retain resp_*; /trajectory renders tool/error/cache/response shape; an append-only live kind:"turn" revision lands before the provider runs and is replaced by the closed revision with the same id; behavior lifecycle remains paired; default operational ev=tool outcomes carry monotonic call_id; content-free first_token timing is emitted; and each executed tool batch checkpoints the durable transcript/session before the next model call. DGM record kinds are unchanged. Verification: 1,702 core tests (1 skipped), 445 TUI tests, all 17 PTY probes, and the full tier-1 release gate passed twice (including the server-side pre-push hook).