Skip to content

Count what each provider actually serves, and name the model that answered each turn - #136

Merged
ParallelEntrepreneur merged 3 commits into
mainfrom
feat/provider-usage-counters
Sep 18, 2026
Merged

ParallelEntrepreneur merged 3 commits into
mainfrom
feat/provider-usage-counters

Conversation

@ParallelEntrepreneur

@ParallelEntrepreneur ParallelEntrepreneur commented Sep 18, 2026 •

Copy link
Copy Markdown
Collaborator

#122 cherry-picked onto current main, with three trivial conflicts resolved and one compile break fixed that only appears on top of today's main. #122 is superseded by this. Closes #39.

A user wired a local DeepSeek server into Colonizer, saw Settings report it Reachable · 1 ms · 1 model, and concluded the local model was never used. Nothing was broken: the provider was pointed at subagent_model, and colonies almost never spawn subagents, so it legitimately saw almost no traffic. The gateway was the only component that knew anything, and it kept only live gauges — in-flight and queued, both zero whenever nobody is mid-request, which is almost always. "Is this thing ever used?" had no answer in the UI.

This adds the record, and makes the UI say what it knows and no more.

  • Cumulative counters per provider. requests, failures, fallbacks, duration_ms, last_request_at, backed by atomics with a Timed RAII guard so a panic mid-request still counts. They survive a restart in provider-usage.json under the data directory, written back when dirty by a flush loop; a missing or corrupt file means the counters start over, never that the gateway fails to build.
  • The model that answered is named per turn. The chat shows the model that did the substantive work, with +N and a tooltip when more than one was involved.
  • Settings reports what it measured rather than implying traffic that never happened.

What changed on the way

Conflict Resolution
main.rs startup tasks Kept both spawns — version::run (the update check, merged since) and gateway::flush_loop
ChatPanel.tsx Kept both additions; they are independent
SettingsDialog.tsx imports Union of the two lists, so meshBroken (#128) and formatDuration (this branch) both survive

One break git could not see. Gateway::new() grew a data_dir argument on this branch, and main.rs has a test helper — added on main after this branch was cut — that still called it with none. The cherry-pick merged cleanly and then failed to compile:

error[E0061]: this function takes 1 argument but 0 arguments were supplied
   --> crates/colonizer/src/main.rs:621:22

Fixed here by passing the helper's own temp root. Worth noting because a textual merge would have produced exactly the same break with no conflict to warn anyone.

Testing

cargo test -p colonizer-harness — 180 pass (three new). npx tsc --noEmit clean, npm run build succeeds. The web suite has no npm test script yet; #132 adds one.

…wered each turn

Refs #39

Co-Authored-By: Colonizer <noreply@colonizer.dev>
#136 makes TurnSummary.models required, which `tsc --noEmit` caught against
the fixtures the CI tests (#138) landed with: neither knew the field. The
fixtures carry it now, and the diff-against-the-previous-turn rule it exists
for has tests of its own — the models a turn actually served, biggest share
first, and nothing named when no total grew.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@ParallelEntrepreneur
ParallelEntrepreneur merged commit 0b3eaa2 into main Sep 18, 2026
7 checks passed
@ParallelEntrepreneur
ParallelEntrepreneur deleted the feat/provider-usage-counters branch September 18, 2026 08:44
ParallelEntrepreneur added a commit that referenced this pull request Sep 18, 2026
#136 rewired the `openai`-wire response path to count what each provider
serves; this branch rewired the same path to price what a colony spends.
Both now run on one path: `openai_response` takes the provider's counters
and the spend recorder the caller already builds, so a routed response is
counted once and priced once. `pricing` joins main's `Provider` fixtures,
and the protocol doc lists `pricing` beside `usage` and `used_by`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ParallelEntrepreneur added a commit that referenced this pull request Sep 23, 2026
#136 makes TurnSummary.models required, which `tsc --noEmit` caught against
the fixtures the CI tests (#138) landed with: neither knew the field. The
fixtures carry it now, and the diff-against-the-previous-turn rule it exists
for has tests of its own — the models a turn actually served, biggest share
first, and nothing named when no total grew.
ParallelEntrepreneur added a commit that referenced this pull request Sep 23, 2026
Count what each provider actually serves, and name the model that answered each turn
ParallelEntrepreneur added a commit that referenced this pull request Sep 23, 2026
#136 rewired the `openai`-wire response path to count what each provider
serves; this branch rewired the same path to price what a colony spends.
Both now run on one path: `openai_response` takes the provider's counters
and the spend recorder the caller already builds, so a routed response is
counted once and priced once. `pricing` joins main's `Provider` fixtures,
and the protocol doc lists `pricing` beside `usage` and `used_by`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No way to tell whether a model provider is ever used

1 participant