feat: OTLP trace export from agent runtime

[jaroslawgajewski]

Problem

gh-aw already depends on go.opentelemetry.io/otel (v1.39.0+) but does not
actively instrument execution paths or export traces. The OpenTelemetry
dependency is present but dormant.

Without OTLP export, there is no way to get structured, per-LLM-call span data
from the agent runtime. All observability is limited to post-run aggregate
metadata, which misses the rich execution timeline.

Proposed solution

Activate OpenTelemetry instrumentation for key code paths and allow users to
configure an OTLP exporter endpoint:

# In workflow frontmatter or env
observability:
  otlp:
    endpoint: ${{ secrets.OTLP_ENDPOINT }}

Or via standard OTel env vars:

• OTEL_EXPORTER_OTLP_ENDPOINT
• OTEL_SERVICE_NAME=gh-aw

Spans to instrument

Code path	Span name	Key attributes
Agent execution	gh-aw.agent.execute	workflow, engine, model
Each agent turn	gh-aw.agent.turn	turn_number, tokens, cost
MCP tool call	gh-aw.mcp.call	tool, server, status
Safe-output processing	gh-aw.output.process	type, validation
AWF network request	gh-aw.awf.request	domain, allowed/blocked

Trace hierarchy

Trace: workflow-run-{call_id}
  └─ Span: gh-aw.agent.execute
       ├─ Span: gh-aw.agent.turn (turn 1)
       │    ├─ Span: gh-aw.mcp.call (get_file_contents)
       │    └─ Span: gh-aw.mcp.call (search_code)
       ├─ Span: gh-aw.agent.turn (turn 2)
       │    └─ Span: gh-aw.mcp.call (create_pull_request)
       └─ Span: gh-aw.output.process (safe-output)

Use cases unlocked

• Any OTLP-compatible backend: Langfuse, Datadog, Honeycomb, Jaeger,
  Grafana Tempo, New Relic — all accept OTLP natively.
• Per-call latency profiling: See which LLM call or tool call is slowest.
• Distributed tracing: Correlate gh-aw spans with downstream service spans.
• No vendor lock-in: Standard protocol, any backend, any visualization.

Implementation notes

• The OTLP endpoint must be auto-added to the AWF firewall allowlist.
• Use noop tracer when OTLP is not configured (zero overhead).
• Graceful degradation: exporter errors must never fail the workflow.
• Batching: use the OTel SDK's built-in batch span processor.

[pelikhan]

@strawgate @lpcox

[strawgate]

This is fantastic!

Are you planning to insert the instrumentation markers into the forwarded calls from the mcp proxy? So that the agent spans can be correlated with downstream systems the tool calls touch?

[pelikhan]

@strawgate what's the shape of those markers typically?

[strawgate]

there's some info here modelcontextprotocol/modelcontextprotocol#246

When the agent sends an HTTP request to /mcp/{serverName}, the gateway extracts the traceparent from HTTP headers (if the agent's engine supports it, and you could probably just skip this one) or from params._meta (per MCP conventions). It creates a gateway span, then injects the trace context into params._meta before forwarding via stdio to the MCP server container or via http somewhere else.

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "get-weather",
  },
}

to

{
  "jsonrpc": "2.0",
  "method": "tools/call",
  "params": {
    "name": "get-weather",
    "_meta": {
      "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
      "baggage": "userId=alice,serverNode=DF 28,isProduction=false"
    }
  },
}

The equally annoying part of this will be that the compiler needs to emit a small preamble step in the agent job that generates a trace ID and writes traceparent to $GITHUB_OUTPUT. Then thread that output into jobs that need to make instrumentable calls like the mcp server, threat detection and safe output jobs, as env vars to those services, which then will automatically instrument any http calls, and allow you to emit on like step, etc.

[pelikhan]

@lpcox

[lpcox]

@jaroslawgajewski @strawgate this looks cool and should be pretty straightforward to implement in the gateway logs. carrying info through other processes/jobs will be trickier.