feat(chat): stream the answer as the model writes it

[dovvnloading]

The problem

Responses did not stream. Both chat clients already consumed a real token
stream from their runtime, joined it into a complete string and returned that;
the API runner then sliced the finished string into 80-character pieces and
published those as generation.content_delta events. The tokens arrived on
time and were discarded.

On a local model at a few tokens per second that is a spinner for 60-100
seconds followed by the entire answer at once. README.md has advertised
streaming since the rewrite, and llamacpp/chat_client.py's own module
docstring said the opposite in writing: "Cortex never actually streams tokens
from the model runtime itself ... the SSE 'typing' effect is Cortex chunking the
already-complete response after the fact."

Nothing about the runtimes needed to change. The plumbing was already there.

What changed

Layer	Change
ChatClient	optional on_delta(kind, text), kind in content/thinking
OllamaChatClient, LlamaCppChatClient	call it from the loops that already read the stream; take the streaming path when either cancellation or on_delta is wanted
RoutingChatClient	forwards it, keeping the existing "only forward when set" contract
SynthesisAgent	passes it to the main turn only -- never repair, translation or title
GenerationService	publishes typed content_delta / thinking_delta progress; marks the result streamed
api/routes.py	skips the replay loop for a streamed result, so the answer is never sent twice

A non-streaming engine keeps the replay, so client-side rendering is identical
either way and the deterministic test double still exercises that path.

The two things this had to get right

1. Control blocks must not appear and then vanish.
_parse_and_clean_response strips four things from the finished reply: a
<memory_command> proposal, a <code_execution_request> block, legacy
<memo>/<clear_memory/> tags, and an inline Thinking... ...done thinking.
trace. Streaming raw tokens shows the user text the final answer does not
contain, which then blinks out when the cleaned answer replaces it.

EnvelopeStreamFilter (new) withholds from the first character that could
begin one of those, then drops the block once confirmed or releases the held
text once ruled out. Matching is case-insensitive, because the cleaner's
own patterns are: a model gets tag case wrong often enough that IGNORECASE is
deliberate there, and an uppercase <MEMORY_COMMAND> is still parsed and
executed. Its design rule is that over-showing is visible and permanent while
under-showing self-heals, so anything still held when the model stops is
dropped rather than revealed.

2. Event volume. One SSE event per token is roughly 20x the old replay, and
the job registry retains a bounded number of events per job. Deltas are
coalesced by size or age, checked as each piece arrives so no timer thread is
needed. Measured over a 500-token answer: 100 tokens/second emits 59 events
instead of 500; 5 tokens/second -- a typical local model -- still sends every
token on its own. It is a ceiling on the fast path, not a delay on the slow one.

Review, and what it caught

Six independent reviewers audited the staged diff under separate lenses
(threading, filter correctness, stream semantics, regression surface, privacy,
tests and docs), each required to prove findings by running code rather than
reading.

Five of the six independently found the same real defect, which I had missed:
the filter covered only two of the four patterns the cleaner strips, and only
in lowercase. Demonstrated leaks, each reproduced through the real agent and
client:

• <MEMORY_COMMAND> -- raw JSON containing a proposed memory typed itself out, then vanished; the command was still parsed and executed
• <Code_Execution_Request> -- model-written source appeared, then disappeared
• <memo> -- legacy tag streamed verbatim
• Thinking... ...done thinking. -- the model's entire private reasoning trace streamed into the answer bubble

The same review found my drift-guard test was vacuous: it grepped llm.py for
the two tag names the filter already knew, so it passed by construction and
could never detect drift. It is replaced by a behavioural test that compares
what streams against what _parse_and_clean_response actually returns, across
every pattern and both chunkings. Reverted against the old filter, that test
fails 19 cases.

One latent issue was also closed: streamed was set before the publisher
validated the delta kind, so an engine emitting only an unrecognised kind would
have suppressed the replay having shown nothing.

Verification

./scripts/check.ps1            All 10 checks passed in 108.8s
python -m pytest -q            995 passed
python -m mypy                 Success: no issues found in 80 source files
python -m ruff check ...       All checks passed
tools/generate_contracts.py    --check clean

Each load-bearing test was confirmed by reverting the behaviour it pins and
watching it fail: the no-double-replay test against a forced replay, the
early-arrival test against a narrowed trigger, and the filter's equivalence
tests against the pre-review table.

End-to-end proof through the real SynthesisAgent and OllamaChatClient, with
the reply delivered in 7-character chunks: all four previously-leaking shapes
now stream exactly the cleaned answer, and a gated fake proves deltas arrive
before the model's final chunk.

Compatibility and rollback

Additive. on_delta is optional at every seam and every ChatClient and
GenerationEngine in the tree declares it; the generated API contract is
unchanged (delta payloads ride an already-untyped field). No schema change, no
settings change, no migration. Rollback is a revert of the feature commit.

Known limits

Cancellation still discards the turn: deltas already shown are replaced when
the client reloads on generation.cancelled, and persisting partial work is a
separate change in the API runner and the persistence path. With translation
enabled the original answer streams and the completed event carries the
translation -- deliberate, documented and tested, since live feedback beats a
spinner and translation is off by default. An unclosed block is withheld from
the stream but still appears in the finished answer, because the cleaner keeps
it; stripping it there is a separate fix.

🤖 Generated with Claude Code