📊 cli-proxy token usage impact: preliminary observations

[lpcox]

Summary

The cli-proxy feature (replacing GitHub MCP server with gh CLI commands routed through a local proxy) shows a ~24% reduction in token usage and cost in the initial Smoke Copilot data. This is a single data point — more runs are needed to confirm the trend.

What cli-proxy changes

With cli-proxy enabled (features: cli-proxy: true):

• Before: Agent → GitHub MCP server (tool schemas injected into context) → GitHub API
• After: Agent → gh bash commands → cli-proxy → DIFC proxy → GitHub API
• LLM calls unchanged: Agent → api-proxy (token tracker) → Copilot API

The key savings mechanism: MCP tool schemas are removed from the context window. GitHub MCP tools inject ~22 tool definitions (~500-700 tokens each, ~10-15K total) into every LLM turn. With cli-proxy, the agent uses gh CLI commands via bash instead, which requires no schema injection.

Observed data

Smoke Copilot (only workflow with before/after data)

Metric	Pre-cli-proxy (Apr 7)	Post-cli-proxy (Apr 10)	Change
Tokens/run	~334K	~262K	-21.6%
Cost/run	$0.68	$0.52	-23.5%
I/O ratio	156:1	235:1	Higher (less output per input)
Cache hit rate	42.2%	—	—
Requests/run	~4	5	Similar

Source data:

• Pre-cli-proxy: Report #1768 (Apr 7, 4 runs, avg $0.68/run)
• Post-cli-proxy: Manual artifact analysis of run §24222066741 (Apr 10, 5 requests, 262K tokens, $0.52)

cli-proxy enabled: PR #1820, merged Apr 8

Other cli-proxy workflows (no before/after comparison yet)

• Smoke Services (cli-proxy: true in PR #1862, not yet merged)
• Firewall Issue Dispatcher (cli-proxy: true in PR feat: enable cli-proxy for smoke-services and firewall-issue-dispatcher #1862, not yet merged)

Caveats

1. Limited data: Only 1 post-cli-proxy Smoke Copilot run was analyzed (the analyzer workflow had a bug preventing it from finding the data — see below)
2. Branch difference: The post-cli-proxy run was on a PR branch (chore/upgrade-ghaw-v0.68.0), not main. The gh-aw version upgrade may contribute to the token difference.
3. I/O ratio increased: The higher I/O ratio (235:1 vs 156:1) suggests the agent may be sending more context per request, though producing similar output. This could be a side effect of longer bash command outputs vs structured MCP tool responses.
4. No cache data: The post-cli-proxy run did not expose cache read/write breakdown in the raw JSONL (all via copilot provider), making cache comparison impossible.

Token analyzer data gap (resolved)

The daily token usage reports (#1878, #1818) reported "Smoke Copilot produces no token-usage.jsonl" — this was incorrect. The data existed in the firewall-audit-logs artifact but the analyzer workflow was downloading the wrong artifact name (agent-artifacts).

Fixes:

• PR #1883: corrects the artifact name
• PR #1884: refactors all 4 token workflows to use gh aw logs --json (which handles artifact naming internally)

Once merged, future reports will correctly include Smoke Copilot data, enabling proper trend tracking.

Expected savings at scale

If the ~24% reduction holds across workflows:

Workflow	Current avg cost/run	Projected with cli-proxy	Daily runs	Daily savings
Smoke Copilot	$0.68	$0.52	~4	~$0.64
Build Test Suite	$4.54	~$3.45	~3	~$3.27
Secret Digger	$0.40 (success)	~$0.30	~15	~$1.50

Note: Build Test Suite and Secret Digger do not currently use cli-proxy. These are projections based on the Smoke Copilot observation.

Next steps

• Merge PRs fix: token analyzers look for wrong artifact name #1883/refactor: use gh aw logs for token analysis workflows #1884 to fix token analyzer data collection
• Monitor next 5-7 days of Smoke Copilot reports for trend confirmation
• Merge PR feat: enable cli-proxy for smoke-services and firewall-issue-dispatcher #1862 to enable cli-proxy on Smoke Services and Firewall Issue Dispatcher
• Consider enabling cli-proxy on Build Test Suite ($4.54/run — highest potential savings)
• Investigate whether the I/O ratio increase is a concern or expected behavior

[lpcox]

Update: cli-proxy is now universal (April 18, 2026)

Since this report was written (April 10), cli-proxy is now enabled on all 29 agentic workflows — not just the few that had it at the time. This happened through two mechanisms:

How cli-proxy reached 100% adoption

Mechanism	Workflows	Date	PR
byok-copilot: true (implicitly enables cli-proxy)	22 Copilot-engine workflows	Apr 17	#2056 (pilot), #2058 (bulk)
cli-proxy: true (explicit flag)	7 non-Copilot workflows (Claude, Codex, OpenCode, etc.)	Apr 17	#2058
Total	29 workflows

Key insight: byok-copilot implicitly includes cli-proxy — the compiled .lock.yml files for byok workflows include ghcr.io/github/gh-aw-firewall/cli-proxy container and --difc-proxy-host flags. This means all 22 byok-copilot workflows get cli-proxy's token savings without explicitly declaring cli-proxy: true.

Revised status of "next steps" from original report

Task	Status	Notes
✅ Fix token analyzer data collection (#1883/#1884)	Completed
✅ Enable cli-proxy on Smoke Services	Done via byok-copilot (#2058)
✅ Enable cli-proxy on Firewall Issue Dispatcher	Done via byok-copilot (#2058)
✅ Enable cli-proxy on Build Test Suite	Done via byok-copilot (#2058)	Highest cost workflow ($4.54/run)
⏳ Monitor 5-7 days of token reports	Only 1 day of data (Apr 17→18)	Need more data
⏳ Investigate I/O ratio increase	Not yet investigated

What changed vs the original analysis

The original report tracked 1 workflow (Smoke Copilot) with cli-proxy. Now:

• 22 Copilot workflows get cli-proxy via byok-copilot
• 7 non-Copilot workflows get cli-proxy via explicit features: cli-proxy: true
• 0 workflows remain without cli-proxy

Expected impact

If the ~24% token reduction holds across all workflows, the savings are now applied to the entire workflow fleet, not just Smoke Copilot. The Build Test Suite ($4.54/run, ~3 runs/day) is now included — the single highest-impact target identified in the original report.

Data needed

It's been ~1 day since universal adoption. The daily token usage analyzers (Copilot and Claude) should start capturing before/after data. A follow-up comparison in ~5-7 days would validate whether the 24% reduction generalizes across diverse workflow types.