Skip to content

feat(agent): stamp elapsedMs on durable run events at emission - #3483

Merged
kojiwakayama merged 3 commits into
mainfrom
worktree-feat-emission-elapsed-ms
Aug 8, 2026
Merged

kojiwakayama merged 3 commits into
mainfrom
worktree-feat-emission-elapsed-ms

Conversation

@kojiwakayama

@kojiwakayama kojiwakayama commented Aug 8, 2026 •

Copy link
Copy Markdown
Contributor

Makes run traces measurable: every durable run event carries a run-relative elapsedMs taken when it was emitted.

Retargeted. The first revision of this PR stamped the AG-UI browser encoder. That encoder feeds live SSE to a connected client, and a scheduled run has no client attached — so it would never have reached the database. This revision stamps ConversationRunEventEncoder, which is the encoder whose output is persisted.

The problem

agent_run_event.created_at is DEFAULT now() — the row's database insert time. No event carried an emission timestamp, so nothing downstream could say when an event actually happened.

Not theoretical. A post-mortem of production run 2f7ae3d4 built a phase waterfall on created_at and concluded the run spent 52% of its wall clock stalled mid-generation. It hadn't — the data described when rows were written. Three signatures in the same export are impossible under the runtime's own state machine, and are what exposed it:

  • streaming_input → pending_input → streaming_input inside 60ms, when pending requires 5s of idle
  • TOOL_CALL_ARGS rows beginning 200s after the streaming_input that marks the first delta — tool-input-delta maps straight to ToolCallArgs, so those are the same event
  • a metronomic ~30ms delta cadence at 2.5ms stdev

Why this encoder

It is the one whose output is persisted:

ChatStreamEvent → ConversationRunEventEncoder.encode()
                → flat run-event records
                → normalizeConversationRunEvents
                → flushConversationRunEventBatches → POST → agent_run_event

Stamping happens on the way out of encode() rather than in each case arm, so every emitted record is treated alike — including ones the encoder synthesises itself, such as the terminal result for a provider-executed call the provider never resolved.

Why run-relative, and why monotonic

Elapsed is measured from encoder creation. One encoder spans a whole run — it carries stepCount and the active message across every step, which is why run 2f7ae3d4's five steps share one. So elapsedMs needs no per-attempt anchor, and agent_run.started_at + elapsed_ms reconstructs wall clock for any event.

Monotonic rather than wall clock: Date.now() can step backwards under NTP, which across thousands of events yields negative durations. Injected rather than captured, so timing stays deterministic in tests the way policy.clock already does for the stream-lifecycle runner.

What makes it live

The chunk mirror owns one encoder per run, so installing the clock at createConversationRunChunkMirror is what makes production events carry the stamp. Callers injecting their own encoder choose their own clock, or none — the three mirror tests that assert exact durable events pin an unclocked encoder, since their subject is the mirror rather than timing.

Tests

Red-first:

  • elapsed is stamped and advances with an injected clock
  • no elapsedMs key appears at all without a clock — the guard that keeps this invisible to consumers that don't want it
  • it survives normalization: an oversized delta splits into parts that each keep the elapsed of the event they came from. A stamp that doesn't survive normalization never reaches the API, which is exactly the gap that made the first revision inert

src/agent + src/internal-agents + src/chat: 1,214 passed, 0 failed. Typecheck, deno fmt --check, deno lint clean. docs/api-reference regenerated.

Follow-up

veryfront-api: ALTER TABLE agent_run_event ADD COLUMN elapsed_ms integer, lift it with payload->>'elapsedMs' (it lands as a top-level field, since these records are flat), and expose it alongside sequence — which exists on the table but is absent from both the API response and the CSV export.

Related: veryfront-studio#6345 renames that export's timestamp column to created_at so it stops implying it is event time.

Summary by CodeRabbit

  • New Features

    • Conversation run events can include elapsed time from the start of encoding.
    • Added options to control timestamp behavior, including custom clocks.
    • Elapsed-time metadata is applied consistently to split and synthesized events.
  • Documentation

    • Updated API reference links.
    • Added documentation for encoder options and refined the encoder API reference.

@kojiwakayama
kojiwakayama requested a review from kwakayama as a code owner August 8, 2026 19:39
@coderabbitai

coderabbitai Bot commented Aug 8, 2026 •

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

The conversation event encoder now supports optional monotonic elapsed-time stamping. The chunk mirror enables it by default. Tests cover clocked, unclocked, and normalized events. Public exports and API references were updated.

Changes

Conversation event timing

Layer / File(s) Summary
Encoder timing contract and stamping
src/agent/conversation/run-events.ts, src/agent/index.ts
ConversationRunEventEncoder accepts an optional nowMs clock and adds rounded, nonnegative elapsedMs values. The options type is publicly exported.
Mirror encoder clock wiring
src/agent/conversation/run-chunk-mirror.ts, src/agent/conversation/run-chunk-mirror.test.ts
The default mirror encoder uses performance.now(). Tests use unclocked encoders for deterministic event assertions.
Timing validation and API references
src/agent/conversation/run-events.test.ts, docs/api-reference/veryfront/agent.md
Tests cover relative timestamps, omitted timestamps, and normalization-split events. API links and the options entry were updated.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ConversationRunChunkMirror
  participant ConversationRunEventEncoder
  participant EncodedEvent
  ConversationRunChunkMirror->>ConversationRunEventEncoder: create default encoder with performance.now()
  ConversationRunEventEncoder->>ConversationRunEventEncoder: capture creation timestamp
  ConversationRunEventEncoder->>EncodedEvent: encode and stamp elapsedMs
Loading

Suggested reviewers: kwakayama

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: stamping elapsedMs on durable run events during emission.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch worktree-feat-emission-elapsed-ms

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
src/agent/ag-ui/browser-encoder.ts (1)

69-75: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use the required public-copy form.

Replace Supply with Use. Replace the em dash with ASCII punctuation.

As per coding guidelines, public TypeScript copy must use use for actions and must not use em dash characters.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/agent/ag-ui/browser-encoder.ts` around lines 69 - 75, Update the public
documentation for nowMs to use “Use” instead of “Supply” and replace the em dash
with ASCII punctuation, without changing the option’s behavior or type.

Source: Coding guidelines

src/agent/ag-ui/browser-encoder.test.ts (1)

831-859: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add timed finalization coverage.

These tests only exercise mapRuntimeStreamEventToAgUiBrowserEvents. Add a clocked test for finalizeAgUiBrowserEvents that asserts elapsedMs on the synthesized closing event and RunFinished. This protects the separate finalization stamping path.

As per coding guidelines, behavior changes must include focused tests.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/agent/ag-ui/browser-encoder.test.ts` around lines 831 - 859, Add a
focused clock-controlled test for finalizeAgUiBrowserEvents, using
createAgUiBrowserEncoderState with nowMs and advancing the clock before
finalization. Assert that both the synthesized closing event and RunFinished
receive the expected elapsedMs, covering finalization stamping separately from
mapRuntimeStreamEventToAgUiBrowserEvents.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/agent/ag-ui/chunk-encoder-bridge.ts`:
- Around line 21-22: Update the nowMs documentation in
src/agent/ag-ui/chunk-encoder-bridge.ts:21-22 to describe it as optional and
state that omission preserves unstamped events. Apply the same opt-in wording in
src/agent/ag-ui/runtime-event-encoder.ts:22-26, removing the false
performance.now and deterministic-test default claims; no implementation changes
are needed.

---

Nitpick comments:
In `@src/agent/ag-ui/browser-encoder.test.ts`:
- Around line 831-859: Add a focused clock-controlled test for
finalizeAgUiBrowserEvents, using createAgUiBrowserEncoderState with nowMs and
advancing the clock before finalization. Assert that both the synthesized
closing event and RunFinished receive the expected elapsedMs, covering
finalization stamping separately from mapRuntimeStreamEventToAgUiBrowserEvents.

In `@src/agent/ag-ui/browser-encoder.ts`:
- Around line 69-75: Update the public documentation for nowMs to use “Use”
instead of “Supply” and replace the em dash with ASCII punctuation, without
changing the option’s behavior or type.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 5d4772c4-c57a-4f92-95cd-fc25b017d4fe

📥 Commits

Reviewing files that changed from the base of the PR and between 9d1d832 and 3dc1731.

📒 Files selected for processing (5)
  • docs/api-reference/veryfront/agent.md
  • src/agent/ag-ui/browser-encoder.test.ts
  • src/agent/ag-ui/browser-encoder.ts
  • src/agent/ag-ui/chunk-encoder-bridge.ts
  • src/agent/ag-ui/runtime-event-encoder.ts

Comment thread src/agent/ag-ui/chunk-encoder-bridge.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3dc1731279

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/agent/ag-ui/browser-encoder.ts Outdated
Comment thread src/agent/ag-ui/browser-encoder.ts Outdated
Comment thread src/agent/ag-ui/runtime-event-encoder.ts Outdated
Comment thread src/agent/ag-ui/browser-encoder.ts Outdated
`agent_run_event.created_at` is `DEFAULT now()` -- the row's database insert
time. No event carried an emission timestamp, so nothing downstream could say
when an event actually happened, and every duration derived from an export
described the writer rather than the run.

That is not theoretical. A post-mortem of production run 2f7ae3d4 built a phase
waterfall on created_at and concluded the run spent 52% of its wall clock
stalled mid-generation. It had not. Three signatures in the same data are
impossible under the runtime's own state machine and are what exposed it: a
streaming_input -> pending_input -> streaming_input burst inside 60ms when
pending requires 5s of idle, TOOL_CALL_ARGS rows beginning 200s after the
streaming_input that marks the first delta, and a metronomic ~30ms cadence.

ConversationRunEventEncoder is the right place because it is the encoder whose
output is persisted: ChatStreamEvent -> encode() -> flat run-event records ->
normalizeConversationRunEvents -> flushConversationRunEventBatches -> POST. The
AG-UI browser encoder is a separate lane that feeds live SSE to a connected
client, and a scheduled run has no client attached, so stamping there would
never have reached the database.

Stamping happens on the way out of encode() rather than in each case arm, so
every emitted record is treated alike -- including the ones this encoder
synthesises, such as the terminal result for a provider-executed call the
provider never resolved.

Elapsed is measured from encoder creation. One encoder spans a whole run: it
carries stepCount and the active message across every step, which is why run
2f7ae3d4's five steps share one. So elapsedMs is run-relative and needs no
per-attempt anchor, and `agent_run.started_at + elapsed_ms` reconstructs wall
clock for any event.

Monotonic rather than wall clock: a Date.now() reading can step backwards under
NTP, which across thousands of events yields negative durations. Injected
rather than captured, so timing stays deterministic in tests the way
policy.clock already does for the stream lifecycle runner.

The chunk mirror owns one encoder per run, so installing the clock there is
what makes production events carry the stamp. Callers injecting their own
encoder choose their own clock, or none -- the three mirror tests that assert
exact durable events pin an unclocked encoder, since their subject is the
mirror rather than timing.

A stamp that does not survive normalization never reaches the API, so that is
covered directly: an oversized delta splits into parts that each keep the
elapsed of the event they came from.
@kojiwakayama
kojiwakayama force-pushed the worktree-feat-emission-elapsed-ms branch from 3dc1731 to faa8ec5 Compare August 8, 2026 19:52
@kojiwakayama kojiwakayama changed the title feat(agent): let the AG-UI encoder stamp elapsedMs at emission feat(agent): stamp elapsedMs on durable run events at emission Aug 8, 2026
ConversationRunEventEncoder is re-exported from veryfront/agent but its new
options type was not, so consumers importing through the supported entrypoint
could not name the constructor parameter, and the type was missing from the
generated API reference.

Raised by Codex review against the superseded AG-UI revision of this PR. The
type it named no longer exists, but the same gap applied to the encoder this
PR now changes.
@kojiwakayama

Copy link
Copy Markdown
Contributor Author

Addressed. All five review threads were filed against commit 3dc173127, which stamped the AG-UI browser encoder. That revision has been replaced: the PR now stamps ConversationRunEventEncoder, and none of the files those comments cite are in the diff any more.

Rather than close them as outdated, I checked whether each concern still applies to the new code.

Fixed — the concern carried over.

"Export the new encoder state options type" (Codex, P2). CreateAgUiBrowserEncoderStateOptions no longer exists, but the identical gap applied to its replacement: ConversationRunEventEncoder is re-exported from veryfront/agent while ConversationRunEventEncoderOptions was not. Fixed in 3b8aa1a9a; it now appears in docs/api-reference/veryfront/agent.md.

Checked — does not apply.

"Remove the false default-clock claim" (CodeRabbit) and "Correct the advertised default clock" (Codex, P2). Both were right about the old revision: the JSDoc promised a performance.now default the factories did not implement. Those factories are no longer touched. The new option documents itself as opt-in, and createConversationRunChunkMirror installs the clock for real, so the stated behaviour and the implementation now agree.

"Replace em dashes in the public API comment" (Codex, P1). Correct for the old JSDoc. The new comments in run-events.ts and run-chunk-mirror.ts use ASCII --; grep -nE "—|–" over both files returns nothing.

"Preserve elapsedMs through the AG-UI SSE schemas" (Codex, P2). The sharpest of the five, and worth spelling out because it prompted a check that closes a gap I had flagged as unverified. It was correct that formatAgUiEvent reparses payloads through schemas that would strip an undeclared field. That path is no longer involved, but the same question applies to the durable path, so I traced every hop the stamp has to survive:

Hop Result
normalizeConversationRunEvents survives, covered by a test asserting an oversized delta splits into parts that each keep their source elapsed
API append validation (PublicSupportedAGUIEventSchema) survives, the event schemas are .passthrough()
API read path (toPublicAgentRunEventPayload) survives, default: return payload

I had been flagging "verified in unit tests but not observed reaching Postgres" as an open risk on this PR. Those three checks close the schema half of it. What remains is only observing a live run, which the first export after merge will show.

@kojiwakayama
kojiwakayama enabled auto-merge August 8, 2026 20:37

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/agent/conversation/run-chunk-mirror.ts`:
- Around line 163-169: Add a focused test for createConversationRunChunkMirror
that omits input.encoder, emits a chunk, and verifies the prepared durable event
has a nonnegative elapsedMs, covering the default ConversationRunEventEncoder
clock behavior.

In `@src/agent/conversation/run-events.ts`:
- Around line 63-74: Update the documentation comments for
ConversationRunEventEncoderOptions and the related elapsed-time logic near the
encoder implementation to remove references to internal tables, columns, and
persistence fields such as agent_run_event, created_at, agent_run.started_at,
and elapsed_ms. Describe only the public elapsed-time behavior and its
relationship to encoder creation and run timing.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 6c39b03a-0117-487a-818a-0fa50da0bed7

📥 Commits

Reviewing files that changed from the base of the PR and between 3dc1731 and 3b8aa1a.

📒 Files selected for processing (6)
  • docs/api-reference/veryfront/agent.md
  • src/agent/conversation/run-chunk-mirror.test.ts
  • src/agent/conversation/run-chunk-mirror.ts
  • src/agent/conversation/run-events.test.ts
  • src/agent/conversation/run-events.ts
  • src/agent/index.ts

Comment thread src/agent/conversation/run-chunk-mirror.ts
Comment thread src/agent/conversation/run-events.ts
@kojiwakayama
kojiwakayama disabled auto-merge August 8, 2026 20:43
…a names

Two review findings on the current revision.

The mirror's default encoder was unproven. Changing that default is what makes
production events carry elapsed, but the three existing mirror tests pin an
unclocked encoder to keep their exact-event assertions deterministic, so nothing
exercised the default. Adding a test that omits `encoder` and asserts a finite,
nonnegative elapsed. Verified it fails with "the default encoder must stamp
elapsedMs" when the default clock is removed.

The option's JSDoc explained itself by naming the API service's table and
columns. This package is published, so a consumer reading it was pointed at
storage they cannot see and do not own. Reworded to describe the behaviour they
can act on: a persisted event otherwise carries only the time it was stored,
which tracks the writer rather than the run. The reasoning is unchanged, the
foreign schema names are gone.
@kojiwakayama

Copy link
Copy Markdown
Contributor Author

Both addressed in bdd9968ad.

Add a test for the default mirror encoder (Major). Accepted as written. Changing that default is the thing that makes production events carry elapsed, and the three existing mirror tests pin an unclocked encoder to keep their exact-event assertions deterministic, so nothing exercised it. Added a test that omits encoder and asserts a finite, nonnegative elapsedMs. I checked it has teeth rather than trusting it: removing the default clock makes it fail with the default encoder must stamp elapsedMs, and restoring it passes.

Remove private persistence details from comments (Minor). Acted on, though not for the cited reason. The guideline quoted covers sensitive values, and a table name is not one. The real problem is narrower and worth stating: this package is published, and the JSDoc for a public option explained itself by naming the API service's table and columns, so a consumer reading it was pointed at storage they cannot see and do not own. That is poor API documentation independent of any rule.

Reworded to describe behaviour the caller can act on:

A persisted event otherwise carries only the time it was stored, which tracks the writer rather than the run, so durations built from it do not describe what the agent did.

Same reasoning, no foreign schema names. grep -nE "agent_run_event|agent_run\.started_at|created_at|elapsed_ms" over both files now returns nothing.

src/agent + src/internal-agents + src/chat: 1,214 passed, 0 failed. Typecheck, fmt, lint clean, API reference regenerated.

@kojiwakayama
kojiwakayama added this pull request to the merge queue Aug 8, 2026
@kojiwakayama
kojiwakayama removed this pull request from the merge queue due to a manual request Aug 8, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/agent/conversation/run-chunk-mirror.test.ts (1)

81-103: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Make mirror cleanup unconditional.

Because flushDelayMs is set to 10_000, the mirror can retain delayed flush work. If handleChunk or an assertion throws, execution skips mirror.dispose(). Move the test body into try and dispose the mirror in finally.

Proposed fix
-    await mirror.handleChunk({ type: "text-delta", id: "m1", delta: "hello" });
-
-    const elapsedMs = prepared[0]?.elapsedMs;
-    assertEquals(typeof elapsedMs, "number", "the default encoder must stamp elapsedMs");
-    assertEquals(
-      typeof elapsedMs === "number" && Number.isFinite(elapsedMs) && elapsedMs >= 0,
-      true,
-      `elapsed must be a finite, nonnegative reading, got ${String(elapsedMs)}`,
-    );
-    mirror.dispose();
+    try {
+      await mirror.handleChunk({ type: "text-delta", id: "m1", delta: "hello" });
+
+      const elapsedMs = prepared[0]?.elapsedMs;
+      assertEquals(typeof elapsedMs, "number", "the default encoder must stamp elapsedMs");
+      assertEquals(
+        typeof elapsedMs === "number" && Number.isFinite(elapsedMs) && elapsedMs >= 0,
+        true,
+        `elapsed must be a finite, nonnegative reading, got ${String(elapsedMs)}`,
+      );
+    } finally {
+      mirror.dispose();
+    }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/agent/conversation/run-chunk-mirror.test.ts` around lines 81 - 103, Wrap
the test body after creating the mirror in a try block and move mirror.dispose()
into a finally block so cleanup always runs, including when handleChunk or an
assertion throws. Keep the existing assertions and mirror configuration
unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/agent/conversation/run-chunk-mirror.test.ts`:
- Around line 81-103: Wrap the test body after creating the mirror in a try
block and move mirror.dispose() into a finally block so cleanup always runs,
including when handleChunk or an assertion throws. Keep the existing assertions
and mirror configuration unchanged.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 852f84df-4665-4ffd-8a0f-72c0aaa3dec1

📥 Commits

Reviewing files that changed from the base of the PR and between 3b8aa1a and bdd9968.

📒 Files selected for processing (3)
  • docs/api-reference/veryfront/agent.md
  • src/agent/conversation/run-chunk-mirror.test.ts
  • src/agent/conversation/run-events.ts
🚧 Files skipped from review as they are similar to previous changes (2)
  • src/agent/conversation/run-events.ts
  • docs/api-reference/veryfront/agent.md

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant