fix(agent): surface provider stream failure causes in logs and spans by kwakayama · Pull Request #4550 · veryfront/veryfront-code

kwakayama · 2026-09-21T21:04:05Z

Description

When a provider body read fails mid-stream, the runtime logged only Agent stream error / RuntimeProviderStreamFailure: Provider stream failed, and the chat <model> span recorded exception.type="Error", exception.message="Error". The real cause sat in a private WeakMap. In the production incident behind #4547 it was RangeError: Anthropic partial_json exceeded 4096 deltas, and finding it took replaying captured streams.

Now:

Log: Agent stream error carries errorCauses: up to 4 wrapped errors, where every field is either a fixed classification or an exact allowlisted text. name is an allowlisted error class (else Error/Unknown). code is a known transient network token (e.g. ECONNRESET), otherwise omitted. message is included only when it equals a member of LOGGABLE_CAUSE_MESSAGES, the complete set of texts the Anthropic parser emits with its fixed limits plus the transport's error reading a body from connection. Any other message is withheld as messageRedacted: true. Prompts, model output, customer data, identifiers and hostnames cannot reach the logs.
```
"errorCauses":[{"name":"RangeError","message":"Anthropic partial_json exceeded 4096 deltas"}]
"errorCauses":[{"name":"TypeError","message":"error reading a body from connection"},{"name":"Error","code":"ECONNRESET","messageRedacted":true}]
```
Span: exception.type / status is RuntimeProviderStreamFailure (only for the real wrapper, not for errors that merely use the name), plus error.cause.type (for example RangeError, TypeError, ECONNRESET). Spans carry only this fixed classification, never cause text.
The client-facing SSE error, the run terminal codes and the public error are unchanged.

Related Issue(s)

Fixes veryfront/veryfront-issue-inbox#1615
Part of veryfront/veryfront-issue-inbox#1622

Verification

New src/agent/runtime/provider-stream-failure-diagnostics.test.ts (3 steps, written to fail first):
- the log includes the cause, and the SSE body does not;
- a nested cause with a code is redacted and capped;
- the span type and error.cause.type are set, with no cause text on any span.
src/observability/telemetry-error.test.ts gained 3 steps (35/35 pass).
Related suites pass: src/observability/tracing (316 steps), agent-span-error-redaction, chat-stream-handler, runtime-stream-cancel, provider-replay-emission, runtime-bridge, retry-policy, src/utils/logger.
deno fmt --check, deno lint and deno check pass on the changed files, and so does lint:module-boundaries.

Notes

error.cause.type is added in the shared span failure path, so any span failing with a standard cause gets it too. It is always a fixed label.
The workflow retry telemetry label and the tool span error.type now also say RuntimeProviderStreamFailure instead of Error. This affects telemetry only.

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation update
Code refactoring
Performance improvement
Test update

Checklist

I have made corresponding changes to the documentation (if applicable)
I have added tests that prove my fix is effective or that my feature works

A provider body read that fails mid-stream is wrapped as RuntimeProviderStreamFailure with a private cause, so the server log and the chat span reported only the wrapper. Log a bounded, credential-redacted cause chain (name, message, code) as errorCauses on "Agent stream error", classify the wrapper as RuntimeProviderStreamFailure on spans, and record the cause class as error.cause.type. The client SSE error is unchanged. Refs veryfront/veryfront-issue-inbox#1615

chatgpt-codex-connector · 2026-09-21T21:04:10Z

You have reached your Codex usage limits for security reviews. Please try again later.

greptile-apps

Your trial has ended. Reactivate Greptile to resume code reviews.

chatgpt-codex-connector · 2026-09-21T21:04:16Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-21T22:18:09.414196Z	`3632eca`	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

coderabbitai · 2026-09-21T21:04:28Z

Understand this PR’s impact

Explore downstream dependencies and potential security impact with Blast Radius.

View blast radius →

Warning

Review limit reached

Next included review available in 24 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 22163746-b9db-47e2-9797-6ac5c5fe35cb

📥 Commits

Reviewing files that changed from the base of the PR and between dac7fd3 and 3632eca.

📒 Files selected for processing (6)

extensions/ext-llm-anthropic/src/anthropic-stream.test.ts
extensions/ext-llm-anthropic/src/anthropic-stream.ts
src/agent/runtime/provider-stream-failure-diagnostics.test.ts
src/observability/telemetry-error.test.ts
src/observability/telemetry-error.ts
src/provider/runtime-loader/provider-http.ts

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: 4fd21ba7-7ee9-4d34-90fc-b2d8161d413d

📥 Commits

Reviewing files that changed from the base of the PR and between 4768a33 and dac7fd3.

📒 Files selected for processing (5)

src/agent/runtime/index.ts
src/agent/runtime/provider-stream-failure-diagnostics.test.ts
src/observability/telemetry-error.test.ts
src/observability/telemetry-error.ts
src/observability/tracing/otlp-setup.ts

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

Walkthrough

The change adds wrapped-cause classification and bounded cause summaries for provider stream failures. Agent runtime logs now include sanitized causes. OpenTelemetry spans record cause types. Regression tests cover client responses, logs, and spans.

Changes

Provider stream diagnostics

Layer / File(s)	Summary
Cause classification and summarization `src/observability/telemetry-error.ts`, `src/observability/telemetry-error.test.ts`	Provider stream failures receive a branded telemetry type. New helpers read safe cause data, classify cause types, and create bounded, redacted, cycle-safe cause summaries.
Span cause attributes `src/observability/tracing/otlp-setup.ts`	Span failure reporting sets `error.cause.type` when cause classification returns a value. Classification failures do not interrupt failure reporting.
Runtime failure logging and regression coverage `src/agent/runtime/index.ts`, `src/agent/runtime/provider-stream-failure-diagnostics.test.ts`	Agent stream error logs include summarized causes. Tests verify generic client errors, sanitized and bounded log causes, and OpenTelemetry error attributes.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant AgentRuntime
  participant ProviderStream
  participant TelemetryError
  participant Logger
  participant OpenTelemetry
  AgentRuntime->>ProviderStream: consume stream
  ProviderStream-->>AgentRuntime: provider stream failure
  AgentRuntime->>TelemetryError: classify and summarize cause
  TelemetryError-->>AgentRuntime: failure type and errorCauses
  AgentRuntime->>Logger: record Agent stream error
  AgentRuntime->>OpenTelemetry: record error status and cause type
  AgentRuntime-->>ProviderStream: expose generic Provider stream failed response

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 70.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 10 functions across 5 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and concisely describes the main change: exposing provider stream failure causes in logs and spans.

✨ Finishing Touches 💡 1

📝 Generate docstrings 💡

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-21T21:04:30Z

📦 Client bundle boundary

Entrypoint	Modules	Source size	Server leaks
`src/index.client.ts`	291	2329 KiB	✅ 0

A server module in a client graph aborts hydration in the browser. New leaks fail CI; known leaks are tracked in scripts/lint/client-bundle-baseline.json to burn down.

gitar-bot · 2026-09-21T21:04:56Z

Gitar is working

_Gitar

kwakayama · 2026-09-21T21:05:30Z

Code Review: 88/100 — Good, minor suggestions

Focused, well-tested fix for a real incident (#4547): surfaces the previously-private cause of RuntimeProviderStreamFailure in logs/spans without leaking it to the client SSE stream.

Strengths

Narrowly scoped diff (5 files) with a clear incident-driven description and before/after log examples.
Security posture is careful and consistent with the file's existing conventions: readErrorCause/summarizeErrorCausesForLog never invoke accessors (own-property-descriptor checks only), redact URL credentials via sanitizeUrlCredentials, cap messages at 500 chars, drop stacks entirely for causes, and walk the cause chain cycle-safely capped at 4 links.
Correctly reuses the existing provider-stream-error-provenance.ts WeakMap boundary rather than exposing the private cause on the public error object — preserves the original security invariant while adding an escape hatch for diagnostics.
Good test coverage: targeted unit tests in telemetry-error.test.ts for redaction, cycle-safety, and rejecting name-lookalikes (Object.assign(new Error(), {name: "RuntimeProviderStreamFailure"}) correctly classified as Error, not the real wrapper), plus an end-to-end test wired to a real OTel BasicTracerProvider asserting the cause text never appears in any span attribute or event.

Concerns

telemetryErrorType now returns RuntimeProviderStreamFailure for all call sites, not just the stream-error log path — this also changes the workflow retry telemetry label and tool span error.type from Error to the new value. The PR notes disclose this, but it's worth confirming no existing dashboard/alert keys off the literal "Error" value for those two telemetry dimensions before merging.
Minor/non-blocking: the cause-walking logic in summarizeErrorCausesForLog (own-descriptor checks, Set-based cycle guard) closely parallels the pattern already in sanitizeStructuredValue's seen handling — not a problem, just a second bespoke implementation of a similar idea in the same file.
CI was still in progress at review time (several jobs in_progress); worth confirming green before merge, particularly ci (lint)/ci (typecheck) given the new imports.

No blocking issues found; the redaction/bounding logic is sound and the tests genuinely exercise the failure mode described in the PR.

Generated by Claude Code

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: dac7fd303e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

codecov · 2026-09-21T21:12:07Z

Codecov Report

❌ Patch coverage is 90.19608% with 10 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/observability/telemetry-error.ts	89.87%	5 Missing and 3 partials ⚠️
...tensions/ext-llm-anthropic/src/anthropic-stream.ts	66.66%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

…tics

chatgpt-codex-connector · 2026-09-21T21:13:28Z

You have reached your Codex usage limits for security reviews. Please try again later.

greptile-apps

Your trial has ended. Reactivate Greptile to resume code reviews.

github-actions · 2026-09-21T21:13:40Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3e0352760e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-09-21T21:20:32Z

You have reached your Codex usage limits for security reviews. Please try again later.

greptile-apps

Your trial has ended. Reactivate Greptile to resume code reviews.

github-actions · 2026-09-21T21:20:47Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ed361697e3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…d transient codes

chatgpt-codex-connector · 2026-09-21T21:26:50Z

You have reached your Codex usage limits for security reviews. Please try again later.

greptile-apps

Your trial has ended. Reactivate Greptile to resume code reviews.

github-actions · 2026-09-21T21:27:01Z

@codex review

chatgpt-codex-connector · 2026-09-21T21:32:27Z

You have reached your Codex usage limits for security reviews. Please try again later.

greptile-apps

Your trial has ended. Reactivate Greptile to resume code reviews.

github-actions · 2026-09-21T21:32:36Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d20b46ff4e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

…ic stream bounds

chatgpt-codex-connector · 2026-09-21T21:39:25Z

You have reached your Codex usage limits for security reviews. Please try again later.

greptile-apps

Your trial has ended. Reactivate Greptile to resume code reviews.

github-actions · 2026-09-21T21:39:34Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3632eca327

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

sonarqubecloud · 2026-09-21T21:52:03Z

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
92.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

kwakayama · 2026-09-21T22:13:34Z

@codex review

chatgpt-codex-connector · 2026-09-21T22:18:08Z

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 3632eca327

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

kwakayama requested a review from kojiwakayama as a code owner September 21, 2026 21:04

kwakayama enabled auto-merge September 21, 2026 21:04

greptile-apps Bot reviewed Sep 21, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/agent/runtime/index.ts

fix(agent): log cause messages only for allowlisted framework diagnos…

3e03527

…tics

greptile-apps Bot reviewed Sep 21, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/observability/telemetry-error.ts Outdated

fix(agent): allow only fully enumerated cause messages in logs

ed36169

greptile-apps Bot reviewed Sep 21, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/observability/telemetry-error.ts Outdated

Comment thread src/observability/telemetry-error.ts Outdated

github-code-quality Bot found potential problems Sep 21, 2026

View reviewed changes

Comment thread src/agent/runtime/provider-stream-failure-diagnostics.test.ts Fixed

fix(agent): log only exact diagnostic messages, allowlisted names, an…

cce61ec

…d transient codes

greptile-apps Bot reviewed Sep 21, 2026

View reviewed changes

test(agent): drop unused assert import

d20b46f

greptile-apps Bot reviewed Sep 21, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/observability/telemetry-error.ts

fix(provider): keep the original error as cause when wrapping Anthrop…

3632eca

…ic stream bounds

greptile-apps Bot reviewed Sep 21, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Sep 21, 2026

View reviewed changes

Comment thread src/agent/runtime/index.ts

kwakayama added this pull request to the merge queue Sep 21, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 21, 2026

kwakayama added this pull request to the merge queue Sep 21, 2026

Merged via the queue into main with commit 74bec82 Sep 21, 2026
62 checks passed

kwakayama deleted the fix/provider-stream-failure-cause branch September 21, 2026 23:28

This was referenced Sep 22, 2026

release: veryfront 0.1.1262 #4554

Closed

chore: open the 0.1.1262-rc prerelease line #4555

Merged

release: veryfront 0.1.1262 #4557

Merged

Conversation

kwakayama commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issue(s)

Verification

Notes

Type of Change

Checklist

Uh oh!

chatgpt-codex-connector Bot commented Sep 21, 2026

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

coderabbitai Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review limit reached

Walkthrough

Changes

Sequence Diagram(s)

❌ Failed checks (1 warning)

Uh oh!

github-actions Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Client bundle boundary

Uh oh!

gitar-bot Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kwakayama commented Sep 21, 2026

Code Review: 88/100 — Good, minor suggestions

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

codecov Bot commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

chatgpt-codex-connector Bot commented Sep 21, 2026

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Sep 21, 2026

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 21, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot commented Sep 21, 2026

Uh oh!

greptile-apps Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 21, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 21, 2026

kwakayama commented Sep 21, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 21, 2026 •

edited

Loading

coderabbitai Bot commented Sep 21, 2026 •

edited

Loading

github-actions Bot commented Sep 21, 2026 •

edited

Loading

gitar-bot Bot commented Sep 21, 2026 •

edited

Loading

codecov Bot commented Sep 21, 2026 •

edited

Loading