Conversation memory, and a wake that doesn't replay the day - #4
Open
ASHR12 wants to merge 8 commits into
Open
Conversation
The two-line preview on a task card showed the model's Markdown literally — "**bold**" and "## heading" — while opening the same card rendered it correctly. The preview is a clamped inline box, so it cannot host headings, lists, or tables; it needs one continuous run of text. Flattens the snippet instead: fences, links, rules, table separators, and line-leading markers are stripped, emphasis is unwrapped only where it wraps non-space text on a word boundary so snake_case names survive, and table pipes become spaces so columns do not run together. The opened reader still gets the real Markdown. Co-authored-by: Cursor <cursoragent@cursor.com>
A Live session is not a place to keep memory. The API re-bills the whole context every turn, native audio accrues ~25 tok/s even in silence, and audio-only sessions are cut off at 15 minutes unless the window is compressed. Iris was running uncompressed, so an all-day session grew without bound and every resume rehydrated more history than the last. Compresses the window at 40000 tokens down to 16000, and moves long-term recall to disk. The sizing is deliberate: tool schemas, the instruction set, and the USER/MEMORY snapshot total ~5.5k tokens and are never pruned by slidingWindow, so a target too close to that floor leaves no room for a Google Search result to land. That is what broke grounded search on the earlier 16384/8192 attempt. Verified live before shipping — a forced compression at 40000/16000, then a search that still returned a real grounded figure; a 4000/2000 control fails the request outright. Conversation memory now lives in ~/.iris/journal/YYYY-MM-DD.md, one section per wake-to-sleep cycle, each closed with a short digest written in Iris's own voice so it reads as memory rather than someone else's notes. Turns are buffered and flushed on a timer, and the digest call happens after the socket closes, so neither touches the audio path. Today's digests are injected at connect; older days go through a new search_conversation tool, which is a local scan over digest lines at ~0.06ms per query. Sessions are detached synchronously before the digest is generated, and digests are inserted positionally, so a wake arriving mid-summary cannot have a late digest misfiled into it. On quit there is no time for a round trip, so the heuristic summary is written inline rather than losing the session. Wake latency: a standby handle refresh could hold a wake for tens of seconds while it connected and polled. A wake now yields to it for at most 1.5s, then closes its socket and takes over — the handle already in hand is valid either way. The refresh nudges immediately instead of after 4s, cutting its worst case from ~27s to ~16s. The microphone opens in parallel with the Gemini connect, because in series the device's startup delay landed after "I'm back" and the first thing said went into a mic that was not listening. GoAway is now acted on early during a silent moment instead of being cut off mid-sentence, leaving busy connections to the existing reconnect path. Also carries three smaller fixes that were pending on this branch: - Farewells no longer guess the time of day. Iris has no clock, so "go to sleep" could be answered with "good night" mid-afternoon. The tool response, the sleep rule, and the session-start greeting now agree. - Gesture control is a remembered preference. The camera no longer switches itself on at wake, in HUD mode, or with the Neural Map; it is off until turned on and stays as left. - ~/.iris/session-log.jsonl records connects, resumes, rejected handles, and refresh takeovers. There was previously no history of any of this. Co-authored-by: Cursor <cursoragent@cursor.com>
Two bugs found by using the app for two days, both diagnosed from the session log added in the previous commit. Waking was replaying the whole conversation. Resuming hands the session back to Google to rehydrate before the model will speak, measured at ~140ms per turn of history: 1.0s fresh, 3.5s at 10 turns, 6.4s at 30, 9.6s at 60. A 40-turn afternoon therefore woke in 15-20 seconds, and every wake was slower than the one before it — for context the journal already holds. Connecting costs ~90ms either way, so a wake now opens a new session and carries the conversation across as text: today's digests plus the last few lines actually spoken. Measured 831ms to first audio, with the previous topic still recalled correctly. Handles now survive only to recover a socket that drops mid-sentence, where the context is already warm; with the journal off, waking falls back to the old replay rather than waking with no memory at all. That also deletes the standby handle-renewal machinery, 139 lines whose only purpose was keeping a handle alive across a nap so a wake could replay into it. It never paid for itself either: across 75 recorded sessions it ran 3 times and collided with a wake 0 times. The previous commit tuned that collision path, which was the wrong target — the log is what showed it never happens. The goodbye loop was self-inflicted. Fixing the "good night in the afternoon" problem expanded the go_to_sleep tool response from one sentence into four, including example sign-offs. That response is the last thing left in the conversation, so every wake read it again as a live instruction and recited another example off the list. Iris spent four consecutive wakes saying "Catch you later" and "Talk soon" to a user who was asking whether she could hear him. The response is one short line again, the time-of-day ban lives only in the system instruction where it does not linger in history, and go_to_sleep is refused in code when nobody has spoken yet this session — a sleep request that predates the user's first word can only be an echo. Prompt wording alone was never going to hold this. Verified against the live API with a context deliberately ending in a farewell: greeted normally, no goodbye, no sleep call, and recalled the prior topic from the journal. Co-authored-by: Cursor <cursoragent@cursor.com>
The day file was headed "2026-07-29" and its sections read "## Session 08:26", so two conversations a minute apart were distinguishable only by a random id, and nothing on the page said which day of the week any of it happened on. Since this is the thing that has to answer "what did we decide on Monday?", that is the wrong shape. Days are now headed with the weekday spelled out, sessions are numbered in the order they happened and stamped with the minute they started and the minute they ended, and each digest sits directly under its heading so scrolling a day reads as a list of summaries with the raw lines underneath. File names stay YYYY-MM-DD.md because retention compares them as strings. Session numbers are counted from the file rather than held in memory, so a relaunch mid-day continues the sequence. The spelled-out date also joins the search haystack, which is what makes "on Monday" or "back in July" match at all. Old files still parse. Also stops treating the journal as load-bearing. Waking with it off was falling back to replaying the session — the slow path this branch exists to remove — on the reasoning that slow continuity beats none. But switching memory off and then getting no memory is the expected outcome, and it should not cost a 15-second wake to arrive at. Off now means a one-second wake that starts each session knowing nothing of the earlier ones, which is also what the setting now says it does. Co-authored-by: Cursor <cursoragent@cursor.com>
The two halves of the screen were not the same conversation. Work restored on launch, because tasks live in Hermes and are fetched back by session id. The talking did not: Comms was React state capped at 40 lines, so pausing or stopping kept it only because the process was still alive, and quitting threw the day away. The journal wrote every turn to disk but nothing ever read it back into the UI, and nothing recorded which Hermes chat a conversation belonged to. A conversation is now identified by that chat. Every turn is indexed in ~/.iris/conversations.db keyed by the thread it was spoken in, and the panel restores from it exactly as the Work Stream restores cards — same id, both halves, so reopening the app tomorrow brings back the whole conversation. Scrolling to the top pages further back across days, with dividers where the day changes. Switching chats re-scopes what Iris remembers as well as what is displayed; leaving memory global would let her answer one project out of another's context, which would make the link cosmetic. SQLite rather than more Markdown parsing: a day is already ~275 turns and a thread runs for weeks, so restoring and paging are queries, not file reads — over the journal each scroll would parse and filter up to 90 day-files. node:sqlite is part of Electron 42, so this costs no native module and nothing to rebuild at package time, verified in the signed bundle. The Markdown stays as the artifact you own and can grep; the index is fed from the journal's own flush, so there is one write path and the two cannot disagree about where a session begins and ends. Existing journals import once at launch (39 sessions, 513 turns, 10ms). Raw retention goes 7 → 30 days, because scrollback promising history the pruner deletes would be a lie. Two bugs worth naming, both found by probing the running app rather than reasoning about it. The IPC handler took the invoke event as its payload, so every page returned the newest 60 turns again. And paging on row id was wrong the moment a backfill inserted whole days out of order: the cursor is now (at, id), with the id only breaking ties, since imported turns are accurate to the minute and share timestamps. Co-authored-by: Cursor <cursoragent@cursor.com>
Picking a chat in the switcher left Comms showing the same conversation whatever you picked. The cause was the fallback I added a commit ago: a chat with no turns of its own borrowed the conversations imported from journals written before threads were tracked. Since no chat had any turns yet, every chat borrowed the same history, so switching appeared to do nothing. The fallback is gone from both the panel and the memory seeded at wake. A chat is now shown strictly its own turns, and a chat nobody has spoken in shows an empty panel and wakes cold — the honest answer, where the alternative is another chat's conversation wearing this one's name. That also removes the backfill it existed to serve, along with the journal parser written for it. Unattributable turns would now be invisible in every chat, so importing them would only refill the index with rows nothing can display. The Markdown journal still holds those days for reading; the index starts from conversations that know where they belong. Verified against the running app: the pinned chat restores its own two seeded turns and no others, a second chat returns only its own, and a chat with nothing said in it returns nothing. Co-authored-by: Cursor <cursoragent@cursor.com>
Answering "yes" the moment Iris finished reading a brief could be rejected as an interrupted read-back, and restaging met the same rejection, so the same brief was read five times in a row while the user kept confirming it. The gate judged the read-back on evidence that was not about the read-back. An interruption belonging to the turn before the proposal was staged voided a brief that had not yet been spoken, and "how much was read aloud" was measured with the Comms transcript buffer, which empties on a debounce and so reported a fully read brief as barely started. The gate now counts the model speech produced since the proposal was staged and ignores an interruption that arrives before a word of it has been read. Submitting still requires a read-back that completes and an answer that follows it. Refusals are logged, since the gate previously left no trace. Co-authored-by: Cursor <cursoragent@cursor.com>
Speaking over Gemini cancels its turn and every tool call still in flight. The tool coordinator skipped all of them, including the call that records what the user is about to confirm, so the confirmation arrived to an empty gate and was refused with "no active proposal" — five times on one brief, with nothing ever reaching Hermes. Staging now survives the cancellation that discards the rest. A cancelled call still gets no response on the wire, so the model is never told the id of a brief staged that way; the gate tracks that and accepts a submit or discard that cannot quote it. An id the model was given must still match, and the read-back and the user's answer are still required. Staging, dispatch, and cancelled tool calls are now recorded in the session log, which held no evidence of whether a brief had ever been staged. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
A Live session is not a place to keep memory. The API re-bills the whole context every turn, native audio accrues ~25 tok/s even in silence, and audio-only sessions are cut off at 15 minutes unless the window is compressed. Iris was running uncompressed, so an all-day session grew without bound — and every wake replayed it.
What changed
Bounded live window. Compression at 40000 → 16000 tokens. Sized against the ~5.5k-token floor of tool schemas, instructions and the
USER/MEMORYsnapshot, whichslidingWindownever prunes — a target too close to that floor leaves no room for a Google Search result to land, which is what broke grounded search on the earlier 16384/8192 attempt.Conversation journal.
~/.iris/journal/YYYY-MM-DD.md, one section per wake-to-sleep cycle, each closed with a short digest in Iris's own voice. Turns are buffered and flushed on a timer; the digest call happens after the socket closes. Neither touches the audio path. Local only — the sole thing sent to Google is the one-shot summarization call.Waking no longer replays the conversation. This is the substantive change, and it came out of two days of real use. Resuming hands the session back to Google to rehydrate before the model will speak:
Roughly 140ms per turn, so a 40-turn afternoon woke in 15-20 seconds and every wake was slower than the last. Connecting costs ~90ms either way, so a wake now opens a new session and carries the conversation across as text — today's digests plus the last few lines spoken. Measured 831ms to first audio, with the previous topic still recalled. Handles survive only to recover a socket that drops mid-sentence. With the journal disabled, waking falls back to the old replay rather than waking with no memory.
That also removed 139 lines of standby handle-renewal machinery, which existed purely to keep a handle alive across a nap so a wake could replay into it. Across 75 recorded sessions it ran 3 times and collided with a wake 0 times.
The goodbye loop, which was self-inflicted. Fixing "good night in the afternoon" expanded the
go_to_sleeptool response from one sentence to four, including example sign-offs. That response is the last thing left in the conversation, so every wake re-read it and recited another example:The response is one short line again, the time-of-day ban lives only in the system instruction, and
go_to_sleepis now refused in code when nobody has spoken yet in the session — a sleep request predating the user's first word can only be an echo. Prompt wording alone was never going to hold this.Diagnostics.
~/.iris/session-log.jsonlrecords every wake with its measured connect time. Both bugs above were diagnosed from it; neither was visible before.Also rides along: gesture control as a remembered preference rather than something that switches itself on, and Markdown no longer leaking into Hermes card previews (separate commit).
Verification
Everything measured against the live API rather than reasoned about, since an earlier attempt at the compression regressed Google Search:
Test plan
search_conversationconnectMsin~/.iris/session-log.jsonlstays flat as the day goes on