Description
Problem
Nothing in the per-service-call persistence path knows whether a message has already been
persisted. Three code facts compose into the defect:
_harness/_agent.py:615 — every harness agent runs with require_per_service_call_history_persistence=True (not configurable).PerServiceCallHistoryPersistingMiddleware (_sessions.py:791) runs around each model call. _prepare_service_call_context (:826) routes every unattributed message of the call's full message list into input_messages , :621— only provider-attributed context messages are excluded), and after the callHistoryProvider.after_runpersistscontext.input_messages wholesale (:597, store_inputs=True` default).save_messages is a blind append — state["messages"] = [*existing, *messages] (_sessions.py:1118, InMemoryHistoryProvider; the file/Redis providers append the same way). No message-identity check exists at any of the three points.
In a looped run (harness todo loop / any multi-iteration flow where each service call's message list carries the accumulated conversation rather than a delta), every LLM round therefore re-persists the whole conversation-so-far. The store grows superlinearly, and — because the store feeds the next round's projection — the model receives a payload containing the conversation multiple times. Session-state persistence then carries the duplicated store across requests, so later turns start from an already-corrupted baseline.
Observed live (production AG-UI deployment)
Harness agent (todo loop) served over the AG-UI FastAPI endpoint; stateless OpenAI-compatible provider (OpenRouter, Kimi K3); the transport passes the full client transcript as run input on every request. One Plan→Execute conversation, single run of the Plan phase (~25 real messages by its end):
| LLM round |
store size before call |
delta |
| 1 |
9 |
— |
| 2 |
21 |
+12 |
| 3 |
35 |
+14 |
| 4 |
51 |
+16 |
| 5 |
69 |
+18 |
The delta each round ≈ the full conversation size, which itself grows by ~2 messages per round — the signature of "re-append everything, every call". By the next run the store held 76 messages for a ~25-message conversation, and token estimates ran ~3× the real conversation (est. ~80k for ~25k real).
Model-visible consequences (all user-visible, all reproduced in one session):
- Our wire-level pairing sanitizer had to merge 4 → 8 → 13 → 17 consecutive-assistant runs round over round (clean stateless growth adds ~1 mergeable boundary per round; +3–4/round means conversation segments were being re-duplicated into the payload as the run progressed) and dropped up to 9 orphan/duplicate tool results in a single payload.
- The model's own reasoning stream literally contained the token
" duplicate" — it was reasoning about the duplicated transcript it was fed.
- In the Plan loop the model re-oriented every round ("The user message appears again — … likely a re-invocation. Let me check my todo list and history."), burning rounds without progress.
- Immediately after the plan-approval turn the model repeated an already-executed
update_mode + todos_add turn verbatim (identical narration text, identical calls) — from its point of view, the duplicated arrangement of its own prior turn read as not-yet-done.
Why it matters
- Correctness: the model works from a corrupted conversation. Repeating state-mutating tool calls (
update_mode, todos_add) corrupts application state; repeated turns and re-asked questions destroy user trust in agent memory.
- Cost/limits: ~2–3× duplicate tokens per call, compounding per round. Long threads hit context limits and trigger compaction/summarization far earlier than the real conversation warrants — and the summarizer then summarizes duplicated content, amplifying the distortion.
- Silent: nothing logs or fails; tolerant providers accept the duplicated payloads, so the failure shows up only as "the model behaves strangely in long looped runs".
Reproduction guidance
The pathology needs each service call's message list to carry the accumulated conversation (not a delta) while a history provider persists per call:
- Build a harness agent (
create_harness_agent) with any history provider, serve it over the AG-UI endpoint (add_agent_framework_fastapi_endpoint) with a stateless chat client (no service-side conversation storage).
- Drive a conversation whose run loops (open todos in a looping mode so
AgentLoopMiddleware iterates), with the client sending the full transcript per request (the AG-UI default).
- Instrument
HistoryProvider.save_messages (or just log len(state["messages"]) / the store row count) per service call.
Expected (buggy) result: the store grows by ~the full conversation size on every LLM round (the table above), and the next round's projection contains duplicate copies.
Note: a bare create_harness_agent(...).run([...]) without the AG-UI transport did not reproduce in our quick attempt — the loop's per-iteration input is only the nudge message there. The full-conversation-as-input shape that triggers the re-append comes from the hosted/AG-UI path, which is why we recommend reproducing through the endpoint.
Code Sample
Error Messages / Stack Traces
Package Versions
agent-framework-core: 1.11.0, agent-framework-ag-ui: 1.0.0rc8
Python Version
Python 3.12
Additional Context
Suggested fix directions
Any one of these closes the defect:
- Persist-once bookkeeping: stamp messages as persisted (e.g. an
additional_properties marker or a per-session set of persisted identities in provider
state) and skip them in HistoryProvider.after_run / the per-service-call persist. An
attribution stamp would also make _split_service_call_messages route
already-persisted messages to the context bucket instead of input_messages on later
calls — fixing both the store growth and the projection duplication at once.
- Identity-dedup in
save_messages: append only messages whose identity (stable
message_id when present, else role + serialized contents) is not already in the
store. This hardens every provider (in-memory, file, Redis) regardless of caller
behavior.
- Delta-only persist: have the per-service-call middleware persist only messages that
were not part of the previous call's persisted set for the same run (the middleware
already owns a per-call SessionContext; it could diff against what it persisted last
round).
Option 1 (or 1+2) seems most aligned with the existing architecture: the middleware is the
only place that knows "this call already persisted these messages".
Our workaround (app-side, remove when fixed)
ats.compaction.DuplicateExclusionCompactionStrategy — an always-on S0 stage in our
before-phase compaction pipeline: excludes exact-duplicate message copies
(identity = role + serialized contents; annotations don't participate; same tool
name/args with different call_ids never collapse) and removes repeated same-object list
references (an exclusion mark on a shared object would drop both occurrences). It runs
before the compaction trigger check so token measurement sees the deduplicated context,
and again after the ladder for idempotency. It cleans the model-bound projection each
call, but the persisted store keeps growing underneath — only an upstream fix stops the
superlinear write amplification.
Description
Problem
Nothing in the per-service-call persistence path knows whether a message has already been
persisted. Three code facts compose into the defect:
_harness/_agent.py:615— every harness agent runs withrequire_per_service_call_history_persistence=True(not configurable).PerServiceCallHistoryPersistingMiddleware(_sessions.py:791) runs around each model call._prepare_service_call_context(:826) routes every unattributed message of the call's full message list intoinput_messages,:621— only provider-attributed context messages are excluded), and after the callHistoryProvider.after_runpersistscontext.input_messageswholesale (:597,store_inputs=True` default).save_messagesis a blind append —state["messages"] = [*existing, *messages](_sessions.py:1118,InMemoryHistoryProvider; the file/Redis providers append the same way). No message-identity check exists at any of the three points.In a looped run (harness todo loop / any multi-iteration flow where each service call's message list carries the accumulated conversation rather than a delta), every LLM round therefore re-persists the whole conversation-so-far. The store grows superlinearly, and — because the store feeds the next round's projection — the model receives a payload containing the conversation multiple times. Session-state persistence then carries the duplicated store across requests, so later turns start from an already-corrupted baseline.
Observed live (production AG-UI deployment)
Harness agent (todo loop) served over the AG-UI FastAPI endpoint; stateless OpenAI-compatible provider (OpenRouter, Kimi K3); the transport passes the full client transcript as run input on every request. One Plan→Execute conversation, single run of the Plan phase (~25 real messages by its end):
The delta each round ≈ the full conversation size, which itself grows by ~2 messages per round — the signature of "re-append everything, every call". By the next run the store held 76 messages for a ~25-message conversation, and token estimates ran ~3× the real conversation (est. ~80k for ~25k real).
Model-visible consequences (all user-visible, all reproduced in one session):
" duplicate"— it was reasoning about the duplicated transcript it was fed.update_mode+todos_addturn verbatim (identical narration text, identical calls) — from its point of view, the duplicated arrangement of its own prior turn read as not-yet-done.Why it matters
update_mode,todos_add) corrupts application state; repeated turns and re-asked questions destroy user trust in agent memory.Reproduction guidance
The pathology needs each service call's message list to carry the accumulated conversation (not a delta) while a history provider persists per call:
create_harness_agent) with any history provider, serve it over the AG-UI endpoint (add_agent_framework_fastapi_endpoint) with a stateless chat client (no service-side conversation storage).AgentLoopMiddlewareiterates), with the client sending the full transcript per request (the AG-UI default).HistoryProvider.save_messages(or just loglen(state["messages"])/ the store row count) per service call.Expected (buggy) result: the store grows by ~the full conversation size on every LLM round (the table above), and the next round's projection contains duplicate copies.
Note: a bare
create_harness_agent(...).run([...])without the AG-UI transport did not reproduce in our quick attempt — the loop's per-iteration input is only the nudge message there. The full-conversation-as-input shape that triggers the re-append comes from the hosted/AG-UI path, which is why we recommend reproducing through the endpoint.Code Sample
Error Messages / Stack Traces
Package Versions
agent-framework-core: 1.11.0, agent-framework-ag-ui: 1.0.0rc8
Python Version
Python 3.12
Additional Context
Suggested fix directions
Any one of these closes the defect:
additional_propertiesmarker or a per-session set of persisted identities in providerstate) and skip them in
HistoryProvider.after_run/ the per-service-call persist. Anattribution stamp would also make
_split_service_call_messagesroutealready-persisted messages to the context bucket instead of
input_messageson latercalls — fixing both the store growth and the projection duplication at once.
save_messages: append only messages whose identity (stablemessage_idwhen present, else role + serialized contents) is not already in thestore. This hardens every provider (in-memory, file, Redis) regardless of caller
behavior.
were not part of the previous call's persisted set for the same run (the middleware
already owns a per-call
SessionContext; it could diff against what it persisted lastround).
Option 1 (or 1+2) seems most aligned with the existing architecture: the middleware is the
only place that knows "this call already persisted these messages".
Our workaround (app-side, remove when fixed)
ats.compaction.DuplicateExclusionCompactionStrategy— an always-on S0 stage in ourbefore-phase compaction pipeline: excludes exact-duplicate message copies
(identity = role + serialized contents; annotations don't participate; same tool
name/args with different
call_ids never collapse) and removes repeated same-object listreferences (an exclusion mark on a shared object would drop both occurrences). It runs
before the compaction trigger check so token measurement sees the deduplicated context,
and again after the ladder for idempotency. It cleans the model-bound projection each
call, but the persisted store keeps growing underneath — only an upstream fix stops the
superlinear write amplification.