[reliability] Daily Reliability Review - 2026-06-15

[github-actions]

Executive Summary

Telemetry is flowing for github/gh-aw (Sentry org github, project gh-aw, id 4511347087179777) over the last 24h: the spans dataset has 100+ spans across many traces (http.server, gen_ai, default ops). However, the 24h reliability picture is inconclusive at the run level and dominated by an instrumentation gap, not a confirmed operational outage:

• No confirmed timeouts or cancellations — gh-aw.run.status:timeout and :cancelled both return 0 spans (queryable, confirmed zero).
• errors and logs datasets are empty for the window — no captured exceptions or log records to corroborate failures.
• 70 spans carry gh-aw.run.status:failure, but they are all gen_ai job-conclusion spans (exactly 5 per trace × 14 traces) and run.status is applied per job-conclusion span, mixed within a single trace (a representative trace has 5 failure + 1 success). Because the run root outcome is not separately queryable (span.status / OTLP status.code is not present in the indexed spans), these cannot be promoted to "14 failed runs" with confidence.

Verdict: inconclusive runtime outcome + confirmed instrumentation gap. Failing job-conclusion spans are real and worth attributing, but truncation, token-runaway, and run-level pass/fail cannot be assessed today because the fields needed for them are not queryable.

Top Reliability Findings

Priority	Workflow	Problem	Evidence	Next Action
P2	(unattributed — see note)	14 traces each carry 5 gen_ai job-conclusion spans tagged gh-aw.run.status:failure; no timeout/cancelled seen	gh-aw.run.status:failure → 70 spans, all op gen_ai, 5/trace × 14 traces (24h). Rep. trace 8db31bfd... = 67 spans, 5 failure + 1 success	Triage the 14 failure traces in Sentry; correlate gh-aw.run.id to GitHub Actions runs
P3	n/a (exporter)	OTLP status.code / status.message not surfacing as queryable span.status	has:span.status → No results; emit side sets status.code/status.message (send_otlp_span.cjs:333)	Confirm backend maps OTLP span status → span.status; otherwise run-level pass/fail stays unqueryable
P4	n/a (truncation/tokens)	Length-truncation and token-cost cannot be assessed	gen_ai.response.finish_reasons absent over 24h and 7d; finish_reasons:length → 0; has:gh-aw.aic → No results	Verify the always-emit path (send_otlp_span.cjs:2138-2139) and array-attr indexing reach Sentry
P5	n/a (instrumentation)	Wrong/duplicate field naming & missing resource attr	has:gh_aw.workflow_name → 0 (correct name is gh-aw.workflow.name, present); has:service.version → 0 (Sentry release present instead)	Standardize on gh-aw.workflow.name; confirm service.version→release mapping is intended

Attribution note: the available Sentry MCP build exposes only list_events, which renders a fixed field subset (span.op, id, trace, timestamp, project.name) and does not print gh-aw.workflow.name / gh-aw.run.id values, so the failing runs could not be named from the CLI. Filter values are confirmed present (has:gh-aw.workflow.name and has:gh-aw.run.id both return results on the failure spans). Use the Sentry link below to read the workflow names directly.

Representative Traces

View representative traces & query evidence

Representative failure trace (verified for continuity — 67 spans, mixed job conclusions):

• 8db31bfdef5474c87005115b55e2a3c8 — https://github.sentry.io/explore/traces/trace/8db31bfdef5474c87005115b55e2a3c8
  • all spans: 67 (mix of http.server, gen_ai, default)
  • trace:8db31bfd... gh-aw.run.status:failure → 5 spans (all gen_ai)
  • trace:8db31bfd... gh-aw.run.status:success → 1 span (gen_ai)
  • → confirms run.status is per-job-conclusion span and mixed within one run

Other failure traces (24h, 5 failure spans each): e83b6727..., bdca3129..., bc8d70f1..., 8e3d060f..., 889e4ff1..., 3ccfa909..., 37f443a4..., 340c4ed7..., 230aafd3..., 1e440dde..., 1c1d6870..., 0fdf143f..., 0e920b53...

Browse all failure spans in Sentry:
https://github.sentry.io/explore/traces/?query=gh-aw.run.status:failure&project=4511347087179777&statsPeriod=24h&table=span

Companion datasets (explicit):

• errors dataset (24h) → No results
• logs dataset (24h) → No results

Recommendations

1. Smallest fix first — verify finish-reason emission reaches Sentry. send_otlp_span.cjs:2127-2139 always pushes gen_ai.response.finish_reasons (via buildArrayAttr) for jobName === "agent", yet has:gen_ai.response.finish_reasons returns nothing over 7 days. Check whether array-valued span attributes are indexed by this backend (EAP), and whether the agent-conclusion span is actually the one carrying it. Until this works, length-truncation is invisible.
2. Make run-level outcome queryable. Either confirm the backend maps OTLP status.code/status.message (send_otlp_span.cjs:333, :2042) to a queryable span.status, or add a run-root gh-aw.run.status so per-job failure/success on gen_ai spans isn't conflated with run outcome.
3. Triage the 14 failure traces. Correlate each trace's gh-aw.run.id to its GitHub Actions run to determine if the failing job-conclusion spans reflect genuine job failures or expected sub-step conclusions. Start from 8db31bfd....
4. Confirm token-cost indexing. has:gh-aw.aic returns nothing despite the emit side always writing gh-aw.aic as a double (send_otlp_span.cjs:2118-2122); without it, runaway-token detection is impossible.

Notes

View notes, tool limitations, and field inventory

Skill file missing: the instructed skills/otel-queries/SKILL.md does not exist in the repo (only a root SKILL.md and .github/skills/*). The fixed query loop was followed regardless: connectivity → spans → errors/logs → representative-trace verification → emit-side cross-check.

MCP build limitations:

• search_events and get_trace_details are not available in this Sentry MCP build (tools: whoami, find_organizations, find_projects, find_releases, find_teams, find_dsns, list_events, list_issue_events, get_doc, search_docs, get_event_attachment, analyze_issue_with_seer). Fell back to list_events + client-side trace:<id> filtering for continuity verification.
• list_events renders only a fixed field subset; requested fields like gh-aw.workflow.name, gh-aw.run.status, gh-aw.aic are not printed. Presence/values were probed via has: and field:value query filters instead.

Attribute inventory (gh-aw spans, 24h) — confirmed via query filters:

• Present / queryable: gh-aw.workflow.name, gh-aw.run.status (values seen: success, failure), gh-aw.run.id, release, span.duration
• Absent / not queryable: span.status, gen_ai.response.finish_reasons (also absent over 7d), service.version, gh-aw.aic, gh_aw.workflow_name (underscore form — never an emitted name; correct emitted form is gh-aw.workflow.name)

Emit-side cross-check (actions/setup/js/send_otlp_span.cjs, modified 2026-06-15 23:24):

• workflow identity → gh-aw.workflow.name (:1290, :2061) — matches telemetry
• run outcome → gh-aw.run.status from agentConclusion/workflowRunConclusion, values success|failure|timeout|cancelled (:2036-2069) — emitted per job-conclusion span
• OTLP status.code/status.message set (:333, :2042, :2053) — not surfacing as span.status
• finish reasons → emitted only for jobName === "agent", always pushed (:2138-2139) — not observed in telemetry
• release correlation → resource attr service.version (:360) — not queryable; release present instead (backend-dependent mapping, as documented)

Inconclusive by design: no timeout/cancelled runs, empty errors/logs, and the failure signal lives only on per-job gen_ai conclusion spans → run-level failures are not confirmed. Reported as instrumentation gap rather than an outage to avoid inventing failure counts.

References:

• §27582993044 (this review run)
• Sentry failure spans (24h): https://github.sentry.io/explore/traces/?query=gh-aw.run.status:failure&project=4511347087179777&statsPeriod=24h&table=span
• Rep. trace: https://github.sentry.io/explore/traces/trace/8db31bfdef5474c87005115b55e2a3c8

Generated by 🚨 Daily Reliability Review · 157.1 AIC · ⌖ 12 AIC · ⊞ 4.9K · ◷

• expires on Jun 17, 2026, 3:33 PM UTC-08:00

[github-actions]

This issue is being closed as outdated. A newer issue has been created: #39696

View newer issue

---

This action was performed automatically by the Daily Reliability Review workflow.