Smoke OTEL - 30577091039

[github-actions]

Executive Summary

Overall result: FAIL

• Local OTLP write: ✅ pass (current-run span confirmed)
• Sentry: 🔶 inconclusive (query works, recent gh-aw spans visible, current-run spans not yet visible)
• Grafana: ❌ fail (read auth/config broken — no trace query capability)
• Datadog: ❌ fail (write export failed with 404; read MCP has zero tools)

Results Table

Backend	Write Config Present	Write Export Succeeded	Read Config Present	Read Query Succeeded	Overall
Local OTLP	✅	✅	—	—	✅
Sentry	✅	✅	✅	🔶	🔶
Grafana	✅	✅	❌	❌	❌
Datadog	✅	❌	❌	❌	❌

Evidence

Step 1: Local OTLP

• GH_AW_OTLP_ENDPOINTS set, configured backend hosts: o205451.ingest.us.sentry.io, otlp-gateway-prod-eu-west-2.grafana.net, otlp-intake.datadoghq.eu.
• Local mirror found at /tmp/gh-aw/otel.jsonl (1 line) containing span gh-aw.agent.setup with github.run_id=30577091039, service.name=gh-aw.smoke-otel-backends, gh-aw.workflow.name=Smoke OTEL, matching this run's URL https://github.com/github/gh-aw/actions/runs/30577091039.
• /tmp/gh-aw/otlp-export-errors.count = 1; /tmp/gh-aw/otlp-export-errors.jsonl contains: {"host":"otlp-intake.datadoghq.eu","status":404,"reason":"Not Found"} — attributable to Datadog only.
• send_status = pass.

Step 2: Sentry

• sentry whoami succeeded (authenticated as Mara Kiefer).
• find_organizations → org github; find_projects --query=gh-aw → project gh-aw found.
• Query dataset=spans, query="run_id:30577091039" (24h, project gh-aw) → no results.
• Query dataset=spans, query="service.name:gh-aw.smoke-otel-backends" → no results.
• Fallback query dataset=spans (no filter, 24h) → results found: recent gh-aw spans exist, e.g. gateway.request, gateway.backend.execute, mcp.tool_call (trace e379b7f4addd6b3bce8f784e6da4d225, timestamp 2026-07-30T20:00:23Z). View: https://github.sentry.io/explore/traces/trace/e379b7f4addd6b3bce8f784e6da4d225
• dataset=errors (24h) → no results. dataset=logs (24h) → no results.
• No OTLP export errors attributed to Sentry in Step 1 evidence.
• sentry_status = inconclusive (query access works, recent gh-aw spans visible, current-run spans not yet visible — likely ingestion delay).

Step 3: Grafana

• list_datasources succeeded: returned 13 datasources including Tempo traces datasource grafanacloud-ghaw-traces (uid grafanacloud-ghaw-traces, id 10).
• Only 2 tools are exposed by this Grafana MCP configuration: get_datasource, list_datasources — no trace/query tool (e.g. no Tempo search or Loki query tool) is available to run any trace search.
• get_datasource --uid=grafanacloud-ghaw-traces → 403 Forbidden: "You'll need additional permissions to perform this action. Permissions needed: datasources:read".
• No OTLP export errors attributed to Grafana in Step 1 evidence (write path appears fine).
• grafana_status = fail (read path broken: missing query tool + insufficient permissions to even inspect the datasource).

Step 4: Datadog

• datadog --help → 0 tools registered (Commands (0): (tool list unavailable)).
• Direct calls to plausible tool names list_events and search_events → Error [-32602]: unknown tool.
• Step 1 evidence shows an OTLP export error attributable to Datadog: {"host":"otlp-intake.datadoghq.eu","status":404,"reason":"Not Found"}.
• datadog_status = fail (both write export failed with 404 and read MCP exposes no query tools at all).

Failure Analysis

Sentry — current-run spans not visible (inconclusive)

• Failing step: Step 2 exact run_id:30577091039 and service.name:gh-aw.smoke-otel-backends span queries.
• Evidence: No results for run-specific filters; unfiltered fallback query returns other recent gh-aw spans (gateway/mcp tool call spans, not gh-aw workflow spans) from ~the same time window.
• Likely root cause: Ingestion/indexing delay between OTLP export and Sentry's span search index becoming queryable, or the span attribute keys run_id/service.name are not indexed/mapped as searchable fields for this dataset (the fallback spans found are gateway/mcp-server-side spans, not the workflow's own gh-aw.agent.setup span — suggesting the workflow's own span may not have propagated to Sentry's queryable index yet).
• Path: Read path / ingestion timing — not write path (no export errors attributed to Sentry).
• Next step: Re-run the same run_id:30577091039 query a few minutes later; if still absent, verify the field name mapping for github.run_id/gh-aw.run.id in Sentry's span schema and confirm the gh-aw.agent.setup span is actually being exported to the Sentry OTLP endpoint (not just gateway/mcp spans).

Grafana — read query failed (fail)

• Failing step: Step 3, all trace search attempts.
• Evidence: Only get_datasource/list_datasources tools exist in the Grafana MCP surface (no Tempo/trace-search tool); the one read call available (get_datasource) returned HTTP 403 datasources:read permission missing for the configured GRAFANA_SERVICE_ACCOUNT_TOKEN.
• Likely root cause: The Grafana MCP server integration in this workflow does not expose a trace-query tool, and/or the service account token lacks the datasources:read RBAC permission needed even to introspect the Tempo datasource, let alone query it.
• Path: Read path — MCP tool surface + auth/permissions configuration.
• Next step: Grant the GRAFANA_SERVICE_ACCOUNT_TOKEN service account the datasources:read role (and any trace-query scope), and confirm the Grafana MCP server config enables a trace-search tool (e.g., Tempo query) rather than only datasource-metadata tools.

Datadog — write and read both failed (fail)

• Failing step: Step 1 export (404) and Step 4 all query attempts.
• Evidence: /tmp/gh-aw/otlp-export-errors.jsonl → {"host":"otlp-intake.datadoghq.eu","status":404,"reason":"Not Found"}; datadog --help shows 0 tools; direct tool invocations return unknown tool.
• Likely root cause: Write path — the OTLP endpoint path/route for otlp-intake.datadoghq.eu returned 404, indicating a misconfigured or incorrect OTLP ingestion path/API key for the Datadog EU intake, or the endpoint requires a different URL suffix (e.g. /api/v2/otlp/v1/traces) than what is configured in GH_AW_OTEL_DATADOG_ENDPOINT. Read path — the Datadog MCP server for this workflow failed to register any tools, indicating an MCP server startup/auth failure (likely missing/invalid DD_API_KEY/DD_APPLICATION_KEY).
• Path: Both write (endpoint/auth/URL misconfiguration) and read (MCP server configuration/auth).
• Next step: Verify GH_AW_OTEL_DATADOG_ENDPOINT matches Datadog's documented OTLP intake path for the EU site, confirm GH_AW_OTEL_DATADOG_API_KEY is valid; separately check the Datadog MCP server logs/config to see why zero tools were registered (likely DD_API_KEY/DD_APPLICATION_KEY missing or invalid at MCP startup).

Run

https://github.com/github/gh-aw/actions/runs/30577091039

Generated by 🧪 Smoke OTEL · auto · 39.3 AIC · ⌖ 5.23 AIC · ⊞ 10.8K · ◷
_{Comment /smoke-otel-backends to run again
Add label smoke to run again}

• expires on Jul 30, 2026, 2:05 PM UTC-08:00

[github-actions]

This issue was automatically closed because it expired on 2026-07-30T22:05:42.300Z.

Closed by Workflow