diff --git a/README.md b/README.md index f4a9a0c..6e60f87 100644 --- a/README.md +++ b/README.md @@ -67,6 +67,8 @@ ak status read-only dashboard: what's true, what's drifted [--json] [--d ak sync converge to good: upgrade + heal + verify [--dry-run] [--no-upgrade] ak dashboard open the local web dashboard (auto-opens your browser) [--port N] [--no-open] [--live-source 'surface=path'] +ak usage inspect/refresh offline provider account analytics + status | refresh openrouter ak host manage execution hosts, routing, and provider bindings status | pick | refresh | off ak run execute a host-neutral activity pipeline (including explicit OpenCode routes) @@ -91,7 +93,8 @@ What the verbs cover: | **setup** | Installs/updates ruflo + agentic-qe + the **agentdb** CLI globally (handling npm ≥11.17's `allow-scripts` so natives build; agentdb is pinned to ruflo's bundled version so the shared learning store stays coherent), installs the **RuvNet Brain** (an offline knowledge base over the rUv stack, powering the `search_ruvnet` MCP — a ~2 GB one-time download, prompted; skip with `--no-ruvnet-brain`), deploys the token-audit skill, merges the managed guidance blocks into the machine-wide guidance files (`~/.claude/CLAUDE.md`, plus `~/.codex/AGENTS.md` on codex machines), offers one-time MCP registration (user scope, with a tool-family picker), and — inside a repo — initializes the project: sanitized `ruflo init`, absolute memory-path pin, a **verified** store→disk write, statusline footer, and a background daemon with **local-only ($0) workers** (token-spending AI workers stay opt-in behind upstream's machine-wide budget). Project scope triggers on a `.git` directory in the current folder; without one it's skipped with a note. `--project` forces it anyway (e.g. a not-yet-`git init`-ed folder), `--minimal` skips it, `--yes` accepts all prompts (non-interactive), `--no-aqe` / `--no-ruvnet-brain` / `--no-security` disable those subsystems, and `--reconfigure` re-offers MCP registration. `--codex` enables + installs the Codex host during setup (dual-mode; both hosts then run at once), and `--primary-host claude\|codex` picks which host leads (codex implies `--codex`). | | **status** | Per-subsystem ✓/⚠/✗ (versions, the kit's own version, **ruvnet-brain** (present + release drift, or "not installed"), natives (agentdb copies **and** ruflo's own memory runtime — the one `npx ruflo memory` loads — load-tested for a native better-sqlite3, not just the agentdb dirs), **memory-pin** (warns when `CLAUDE_FLOW_DB_PATH` points off the live DB), security, learning, aqe/RVF, **agentdb** (CLI present + coherent with ruflo's bundled version, or a store-skew warning), MCP, **hosts** (claude/codex/opencode version + install method; the Claude/Codex **primary** marked and failed when absent), **providers** (host wiring + aqe fallback chain, or "drifted"/claude-only default), **routing** (the persisted activity host+model policy; only Claude/Codex routes project into AQE), daemons, guidance-file blocks (`~/.claude/CLAUDE.md`, project `AGENTS.md`, and `~/.codex/AGENTS.md` on codex machines), statusline), each drift row naming what `sync` would do about it — plus a **health-history** line that flags regressions since the last sync (learning shrank, native slots dropped, drift/security backslid). | | **sync** | The one convergence verb: upgrades first when a new release exists, then re-heals everything an upgrade wipes, then re-checks and reports. Included in that heal: it **installs any enabled frontier host** (claude/codex/opencode) that's entirely absent — never touching an external (mise/brew/native) install — and **re-applies provider wiring** (the `ENABLE_*` host env, OpenCode's native configuration, the aqe fallback chain, and ruflo API providers) whenever it has drifted — and, on a dual-host project, **seeds/heals the Claude/Codex default routing policy** (materializing eligible routes into agentic-qe's `agentOverrides`, e.g. after an aqe upgrade first makes it eligible). It also **installs/repins the standalone `agentdb` CLI** to ruflo's bundled version (keeping the shared cognitive store coherent) and appends a **health-history snapshot** so `status` can flag regressions across syncs. It also **re-runs the RuvNet Brain installer** to pull the latest release when the on-disk KB has drifted (or installs it if absent, when enabled). It also **self-updates the kit**: when a newer `@pacphi/agentic-kit` exists it installs it as the *last* step (the new code applies from the next `ak` run, never mid-sync). Prerelease installs (`4.0.0-alpha.*`) track the `next` npm dist-tag as well as `latest`, so alphas see their successors; stable installs only ever follow `latest`. `--no-upgrade` skips the self-update along with the package upgrades. | -| **dashboard** | Opens a read-only local web dashboard (`127.0.0.1:7431`, localhost-only, never detaches) with seven tabs: **Overview · Hosts & Routing · Providers · Runtime · Intelligence · Usage · Live**. The first five render `ak status` health and routing; Usage indexes local Claude/Codex transcripts on demand. Live groups work by project, then host-branded root sessions with nested agent/worker threads and independently evidenced provider/model metadata, and pairs an interactive agent/tool execution canvas with a rich, server-masked transcript stream. Active sessions can be followed live or reviewed with synchronized play/pause/seek; completed sessions remain available for bounded playback. Live contains no chat or control plane. Ruflo, agentic-qe, and dual-run stores are not auto-discovered; register each trusted structured JSONL file with repeatable `--live-source 'surface=path'` (`surface` is `ruflo`, `aqe`, or `dual-run`). The page is self-contained and offline-first (no internet fetches; local files and loopback subprocesses/endpoints only). A fresh **per-session token** is minted at startup and required by every `/api/*` route (`x-dash-token` header; the launch URL carries it in the `#` fragment, same contract as `ak admin` — ADR-0007, ADR-0014) — this page serves full transcript text, so it is gated the same way admin already gated GitHub/npm stats. See [Live Sessions](docs/LIVE-SESSIONS.md) for coverage, syntax, and privacy limits. **Auto-opens your browser** (`--no-open` for headless/SSH); `--port N` changes the port; tabs deep-link (`#live`) and persist. Stop with Ctrl-C. (Also available as `ak x dashboard`.) | +| **dashboard** | Opens a read-only local web dashboard (`127.0.0.1:7431`, localhost-only, never detaches) with seven tabs: **Overview · Hosts & Routing · Providers · Runtime · Intelligence · Usage · Live**. The first five render `ak status` health and routing; Usage indexes local Claude/Codex/OpenCode transcripts on demand and can display a separately cached OpenRouter account-analytics block. That block never changes local transcript/session/host totals; the dashboard itself never fetches it. Live groups work by project, then host-branded root sessions with nested agent/worker threads and independently evidenced provider/model metadata, and pairs an interactive agent/tool execution canvas with a rich, server-masked transcript stream. Active sessions can be followed live or reviewed with synchronized play/pause/seek; completed sessions remain available for bounded playback. Live contains no chat or control plane. Ruflo, agentic-qe, and dual-run stores are not auto-discovered; register each trusted structured JSONL file with repeatable `--live-source 'surface=path'` (`surface` is `ruflo`, `aqe`, or `dual-run`). The page is self-contained and offline-first (no internet fetches; local files and loopback subprocesses/endpoints only). A fresh **per-session token** is minted at startup and required by every `/api/*` route (`x-dash-token` header; the launch URL carries it in the `#` fragment, same contract as `ak admin` — ADR-0007, ADR-0014) — this page serves full transcript text, so it is gated the same way admin already gated GitHub/npm stats. See [Live Sessions](docs/LIVE-SESSIONS.md) for coverage, syntax, and privacy limits. **Auto-opens your browser** (`--no-open` for headless/SSH); `--port N` changes the port; tabs deep-link (`#live`) and persist. Stop with Ctrl-C. (Also available as `ak x dashboard`.) | +| **usage** | Reads provider-account analytics from local cache (`ak usage status`) or performs one explicit OpenRouter management-API refresh (`ak usage refresh openrouter`). Refresh requires `OPENROUTER_MANAGEMENT_KEY`, writes a credential-free mode-`0600` cache, and discards endpoint/user/key/session identifiers. `status` and dashboard reads make no network request. OpenRouter account rows have no grounded host/session/project correlation and are never merged into transcript totals. | | **admin** | Opens the **maintainer admin** (`127.0.0.1:7432`, localhost-only, foreground) — the project-telemetry sibling of `dashboard`, with the same dark/light visual theme and persisted theme preference: unique repo visitors and cloners (GitHub traffic API, needs a push-access token via `GITHUB_TOKEN`/`GH_TOKEN`/`gh auth token` — panels degrade honestly without one), contributors and watchers, npm download momentum (last 7d vs prior 7d, sparklines — shown as trend only, never an absolute reach number, since mirrors/CI inflate the raw count), latest CI run status and open Dependabot alerts, a **"since you last looked"** delta strip over a local baseline, open issues/PRs from others (oldest first), and external humans ranked by recency (bots excluded). Access is gated by a **per-session token** carried in the URL fragment and sent header-only; the page makes **zero external fetches** (the server proxies GitHub/npm; your credential never reaches the page or the payload — ADR-0007, ADR-0013). Where `dashboard` is offline-first, `admin` does deliberate GitHub/npm egress — that contract split is why they're siblings, not tabs. `--port N`, `--no-open`; Ctrl-C stops. (Also available as `ak x admin`.) | | **run** | **Canonical execution surface.** Executes the template vocabulary through host-neutral supervised adapters. It accepts an explicit OpenCode route (persisted or `--route`) alongside Claude/Codex; `--dry-run` prints the exact plan (with each worker's escalation ladder); `--escalate` advances a failed worker one rung of its route's ladder per attempt (bounded by the ladder; permission/consent and uncertain results are never escalated — ADR-0019). An OpenCode worker runs an isolated loopback server with ephemeral basic authentication, returns only normalized observed facts, and aborts instead of approving a permission request. `ak run` does not turn OpenCode into an AQE provider or primary host. | | **dual** | **Deprecated compatibility wrapper** for existing Claude+Codex `claude-flow-codex` scripts, including its legacy escalation behavior. New execution work should use `ak run`; `ak dual` warns on stderr and will be removed before the stable release. | diff --git a/bin/agentic-kit.mjs b/bin/agentic-kit.mjs index 9e0e6c6..feed0ed 100755 --- a/bin/agentic-kit.mjs +++ b/bin/agentic-kit.mjs @@ -21,6 +21,7 @@ const PORCELAIN = Object.assign(Object.create(null), { setup: () => import('../src/commands/setup.mjs'), dashboard: () => import('../src/commands/x/dashboard.mjs'), admin: () => import('../src/commands/x/admin.mjs'), + usage: () => import('../src/commands/usage.mjs'), run: () => import('../src/commands/run.mjs'), dual: () => import('../src/commands/dual.mjs'), host: () => import('../src/commands/x/provider.mjs'), @@ -50,6 +51,7 @@ Usage (ak = alias of agentic-kit): ak sync converge to good: upgrade + heal + verify [--dry-run] [--no-upgrade] ak dashboard open the local web dashboard (localhost; auto-opens browser) [--port N] [--no-open] ak admin maintainer-only telemetry admin (localhost; GitHub/npm egress) [--port N] [--no-open] + ak usage inspect/refresh offline provider analytics [status|refresh openrouter] ak run execute a host-neutral activity pipeline [template "task"] [--dry-run] ak dual deprecated compatibility wrapper; use ak run for new work ak host manage agent hosts, routing, and provider bindings [status|pick|refresh|off] @@ -163,7 +165,10 @@ async function main() { // effect of the network call — a real disk write that contradicts // "--dry-run: prints the plan, changes nothing" even though it never // touches an ak-managed path. - if (!values.json && !values['dry-run'] && cmd !== 'sync') { + // `ak usage status` promises a pure offline cache read. The explicit + // `refresh` subcommand owns its one named network request; neither form may + // silently add unrelated npm probes through the generic drift nudge. + if (!values.json && !values['dry-run'] && !['sync', 'usage'].includes(cmd)) { try { const { driftReport } = await import('../src/lib/versions.mjs'); for (const r of await driftReport()) { diff --git a/docs/PROVIDERS.md b/docs/PROVIDERS.md index 5fb9f5a..43e431e 100644 --- a/docs/PROVIDERS.md +++ b/docs/PROVIDERS.md @@ -26,6 +26,14 @@ provider can therefore have independent `ollama-via-claude` and `ollama-via-code OpenRouter is a provider behind a host, never automatically a third host. OpenCode is an opt-in activity-routing host through `ak run`, while remaining ineligible as a primary host or AQE provider. Its configured selector does not establish provider, billing, or vendor-diversity facts. + +**Account analytics is separate from routing evidence.** `ak usage refresh openrouter` explicitly +fetches OpenRouter's supported 30-completed-UTC-day activity view with +`OPENROUTER_MANAGEMENT_KEY` and writes a private local cache. `ak usage status` and the dashboard +read only that cache. Because the management response has no local host/session/project/task +correlation key, its rows appear only as provider account analytics and never alter transcript +totals or prove which host executed a request. + This capability model is [ADR-0016](adr/0016-capability-driven-integration-adapters.md) (Accepted); the controls below implement it and remain backward compatible. diff --git a/docs/TRANSCRIPTS.md b/docs/TRANSCRIPTS.md index 0b515bd..9e13fb1 100644 --- a/docs/TRANSCRIPTS.md +++ b/docs/TRANSCRIPTS.md @@ -336,10 +336,12 @@ Deep links: `#usage/` opens the Transcript view directly | Delegation markers | `isSidechain` → `sidechain` flag | `thread_source: "subagent"` → excluded from aggregation, session kept visible | | Session title | model-written `ai-title`, first-prompt fallback | first prompt clipped | -**Planned provider attribution:** OpenRouter-served sessions are invisible to the -scorecard today; ingesting them (discovery → parser → `kind` attribution → -pricing → by-host UI) is tracked as -[#59](https://github.com/pacphi/agentic-kit/issues/59). +**OpenRouter boundary:** the supported activity API has account-level date/model/provider/token/ +request/spend rows, but no transcript or local-session correlation key. `ak usage refresh openrouter` +caches that account view explicitly; the dashboard renders it under separate provider analytics and +never treats it as a third transcript host. OpenRouter-served inference can still be attributed on a +real OpenCode transcript when OpenCode itself records provider/model/cost evidence. See ADR-0009 §9 +and [#59](https://github.com/pacphi/agentic-kit/issues/59). --- diff --git a/docs/USAGE-SCORECARD-METRICS.md b/docs/USAGE-SCORECARD-METRICS.md index 552cbc1..29f22e3 100644 --- a/docs/USAGE-SCORECARD-METRICS.md +++ b/docs/USAGE-SCORECARD-METRICS.md @@ -68,17 +68,25 @@ The parsers are `parseClaude` (`usage-index.mjs:427-510`) and `parseCodex` (`usage-index.mjs:527-628`). Both are pure functions over the raw file bytes — no network, no clock dependency beyond the transcript's own timestamps — so every downstream number traces back to bytes already on the user's disk. -Nothing in this pipeline calls a provider API or a billing endpoint; **no -metric on this tab is ever a copy of an actual invoice.** That is the whole -reason every dollar figure is labelled "API-equivalent." +Nothing in this transcript pipeline calls a provider API or a billing endpoint; **no transcript +metric is ever a copy of an actual invoice.** That is the whole reason every transcript-derived +dollar figure is labelled "API-equivalent." The current persisted field named `provider` identifies which host transcript parser produced a session row; it is not sufficient evidence of the inference provider. The Proposed model in [ADR-0016](adr/0016-capability-driven-integration-adapters.md) separates host, provider, -projection, observability source, and binding, with provenance attached per field. Until that -migration and issue #59's OpenRouter ingestion land, the Scorecard must not invent an OpenRouter -host, infer provider identity from a Claude/Codex transcript alone, or turn unknown billing into -subscription, metered, local, or `$0`. +projection, observability source, and binding, with provenance attached per field. The migration is +now implemented. The Scorecard must not invent an OpenRouter host, infer provider identity from a +Claude/Codex transcript alone, or turn unknown billing into subscription, metered, local, or `$0`. + +OpenRouter account analytics is a separate evidence class. An explicit +`ak usage refresh openrouter` fetches the supported 30-completed-UTC-day management view into a +mode-`0600` local cache; normal dashboard reads remain offline. `/api/usage` exposes that cache only +as `providerAnalytics.openrouter`, never by adding it to `totals`, `byHost`, `byProvider`, +`byModel`, projects, categories, findings, or sessions. The upstream response has no host/session/ +project/task correlation key, so no join is attempted. OpenRouter-credit usage and BYOK external +inference estimates also remain separate rather than being presented as one spend number. See +ADR-0009 §9. --- diff --git a/docs/adr/0009-usage-scorecard-local-transcript-analytics.md b/docs/adr/0009-usage-scorecard-local-transcript-analytics.md index 682bb42..f992dc5 100644 --- a/docs/adr/0009-usage-scorecard-local-transcript-analytics.md +++ b/docs/adr/0009-usage-scorecard-local-transcript-analytics.md @@ -2,6 +2,8 @@ - **Status:** Accepted - **Date:** 2026-07-25 +- **Updated:** 2026-07-30 +- **Update note:** Added the explicit OpenRouter account-analytics cache boundary for issue #59. - **Deciders:** agentic-kit maintainers ## Context @@ -311,6 +313,41 @@ prompt" and "not the human" are different claims. Codex rollouts record only rea > capability, so a reader can tell "this workload had no cache hits" from "this provider cannot > report cache hits" — which the panel currently renders identically. +### 9. OpenRouter account analytics is explicit, cached, and never session-shaped + +**Amendment (2026-07-30; issue #59).** OpenRouter's supported management endpoint, +`GET /api/v1/activity`, returns account activity grouped by endpoint for the last 30 completed UTC +days. It reports date, model, upstream provider, requests, tokens, OpenRouter usage, and an estimated +BYOK inference amount. It does **not** report a host, local session, project, task, transcript, or +grounded correlation key. + +Treating those rows as a third transcript store would therefore fabricate precisely the relationship +ADR-0016 forbids. The adopted boundary is: + +- `ak usage refresh openrouter` is the only network path. It requires + `OPENROUTER_MANAGEMENT_KEY`; `OPENROUTER_API_KEY` is intentionally not accepted as a substitute. +- Refresh normalizes the response into + `~/.config/agentic-kit/openrouter-activity.json`, written atomically with mode `0600`. Unknown + fields, endpoint IDs, credentials, API-key hashes, user IDs, filters, prompts, and content are not + persisted. Endpoint-grouped rows are aggregated into date/model/upstream-provider buckets. +- `ak usage status` and the dashboard read only that local cache. Dashboard startup, polling, and + Usage-tab scans perform no OpenRouter request. +- The cache records the endpoint's objective 30-completed-UTC-day window separately from the first + and last dates that actually contain activity. Quiet days do not shorten the stated coverage. +- The `/api/usage` response carries the cache only under + `providerAnalytics.openrouter`. The UI labels it **provider account analytics**. It never changes + transcript-derived `totals`, `byHost`, `byProvider`, `byModel`, project/category rows, findings, + session counts, or pricing. +- A missing or malformed cache is an honest empty state with the explicit refresh command. A failed + refresh preserves the last good cache. Every documented ActivityItem field is required and + type-checked; malformed rows fail the whole refresh instead of silently understating totals. +- OpenRouter-credit usage and BYOK external inference estimates remain distinct metrics. They are + never silently added into one spend figure. + +This closes the observable part of issue #59 without pretending the API supplies per-session logs. +If OpenRouter later publishes a stable correlation identifier that a host also records, joining the +two evidence streams requires a new decision and fixtures from both sides. + ## Consequences ### Good @@ -338,6 +375,9 @@ prompt" and "not the human" are different claims. Codex rollouts record only rea - **Codex coverage is thin in practice** (5 sessions on the reference machine, none in-window). The panel reports Codex as present-but-idle rather than inventing data, and the split UI is built now so dual-host users are not a later retrofit. +- **OpenRouter account coverage is bounded to 30 completed UTC days.** That is the supported + endpoint's contract, not a user-selectable transcript window. Account rows cannot explain which + host or task generated them. ## References @@ -350,6 +390,10 @@ prompt" and "not the human" are different claims. Codex rollouts record only rea ADR-0008 (machine vs repo scope). - Spec (archived on completion, per `docs/archive/README.md`): [`docs/archive/2026-07-25-superpowers-spec-usage-scorecard.md`](../archive/2026-07-25-superpowers-spec-usage-scorecard.md). - Existing reporter: `claude/skills/ruflo-token-audit/scripts/ruflo-token-audit.py`. +- [OpenRouter — Get user activity grouped by endpoint](https://openrouter.ai/docs/api/api-reference/analytics/get-user-activity) + (30 completed UTC days; management key required). +- [OpenRouter — Management API keys](https://openrouter.ai/docs/guides/overview/auth/management-api-keys) + (administrative keys are distinct from completion/inference keys). - Grounding for §6: [Anthropic — Introducing Claude Opus 5](https://www.anthropic.com/news/claude-opus-5); [`pacphi/retort` versions-blog](https://github.com/pacphi/retort/blob/main/versions-blog.md); [OckBench (arXiv 2511.05722)](https://arxiv.org/html/2511.05722); diff --git a/docs/adr/0016-capability-driven-integration-adapters.md b/docs/adr/0016-capability-driven-integration-adapters.md index 4f016ae..009fb39 100644 --- a/docs/adr/0016-capability-driven-integration-adapters.md +++ b/docs/adr/0016-capability-driven-integration-adapters.md @@ -3,8 +3,9 @@ - **Status:** Accepted - **Date:** 2026-07-28 - **Updated:** 2026-07-30 -- **Update note:** Added read-only Codex plugin-hook compatibility facts and - runtime-selected Ruflo project-memory store proofs. +- **Update note:** Added read-only Codex plugin-hook compatibility facts, + runtime-selected Ruflo project-memory store proofs, and the non-correlatable + OpenRouter account-analytics boundary. - **Deciders:** agentic-kit maintainers - **Related:** [ADR-0001](0001-one-routing-policy-many-projections.md), [ADR-0003](0003-auto-seed-dual-host-provenance.md), @@ -383,8 +384,10 @@ supports it. It never enters the host registry. A host transcript proves the host. OpenRouter response metadata can prove the provider and exact model. The two evidence streams describe one execution only when correlation is grounded. The -normalized fact/resolution seam is the integration point for issue #59; this ADR does not absorb -that issue's full parser, ingestion, pricing, or scorecard delivery. +normalized fact/resolution seam remains the future integration point for correlated execution +evidence. Issue #59 found that OpenRouter's supported activity endpoint exposes no such correlation +key, so ADR-0009 §9 keeps its explicitly refreshed account analytics separate from transcript and +host totals instead of manufacturing a join. #### OpenCode at initial adoption @@ -426,7 +429,7 @@ not write real home/global configuration. - It does not persist credentials. - It does not make every managed host primary or routable. - It does not make OpenCode routable. -- It does not implement issue #59's full transcript, usage, or pricing work. +- It does not claim that OpenRouter account analytics is a transcript or per-session evidence source. - It does not claim provider provenance from host evidence alone. - It does not add dashboard writes or controls. - It does not silently adopt or overwrite externally managed configuration. diff --git a/src/commands/usage.mjs b/src/commands/usage.mjs new file mode 100644 index 0000000..9c07e9e --- /dev/null +++ b/src/commands/usage.mjs @@ -0,0 +1,126 @@ +// Explicit provider-account analytics refresh + offline cache status. +import { heading, info, ok, warn, dim } from '../lib/output.mjs'; +import { + openRouterActivityFile, + readOpenRouterActivity, + refreshOpenRouterActivity, +} from '../lib/usage-openrouter.mjs'; + +export const options = { + json: { type: 'boolean', default: false }, + 'dry-run': { type: 'boolean', default: false }, +}; + +export const help = `ak usage — provider account analytics cache + +The local transcript scorecard remains automatic and offline in \`ak dashboard\`. +This command manages separately fetched provider-account metadata. Provider +analytics are never merged into transcript-derived sessions, host totals, +projects, or task attribution. + +Usage: + ak usage status + ak usage refresh openrouter + +Environment: + OPENROUTER_MANAGEMENT_KEY required only for refresh; an inference key is + intentionally not accepted + +Options: + --json emit machine-readable cache status/result + --dry-run describe an OpenRouter refresh without network or writes + +Examples: + ak usage status inspect the offline cache; no network + ak usage refresh openrouter explicitly fetch the last 30 completed UTC days + ak usage status --json print the normalized credential-free cache`; + +function summary(value) { + if (!value) return null; + return { + provider: value.provider, + source: value.source, + fetchedAt: value.fetchedAt, + coverage: value.coverage, + totals: value.totals, + models: value.byModel?.length ?? 0, + upstreamProviders: value.byProvider?.length ?? 0, + }; +} + +/** + * @param {{ flags: Record, positionals: string[], + * deps?: { cacheFile?: string, read?: typeof readOpenRouterActivity, + * refresh?: typeof refreshOpenRouterActivity } }} input + */ +export async function run({ flags, positionals, deps = {} }) { + const action = positionals[0] ?? 'status'; + const provider = positionals[1]; + const cacheFile = deps.cacheFile ?? openRouterActivityFile(); + const read = deps.read ?? readOpenRouterActivity; + const refresh = deps.refresh ?? refreshOpenRouterActivity; + + if (action === 'status' && provider === undefined) { + const value = read({ cacheFile }); + if (flags.json) { + console.log(JSON.stringify({ cacheFile, openrouter: value }, null, 2)); + return 0; + } + heading('ak usage — offline provider analytics'); + if (!value) { + info('OpenRouter: no local activity cache.'); + info('Refresh explicitly: ak usage refresh openrouter'); + return 0; + } + const t = value.totals; + ok(`OpenRouter cache: ${value.fetchedAt}`); + info(`${t.requests} requests · ${t.promptTokens + t.completionTokens} tokens · ` + + `${value.byModel.length} model(s) · ${value.byProvider.length} upstream provider(s)`); + info(dim('account-level only · never merged into local session or host totals')); + return 0; + } + + if (action === 'refresh' && provider === 'openrouter' && positionals.length === 2) { + if (flags['dry-run']) { + const plan = { + dryRun: true, + action: 'refresh', + provider: 'openrouter', + cacheFile, + network: false, + writes: false, + }; + if (flags.json) console.log(JSON.stringify(plan, null, 2)); + else { + heading('ak usage — refresh plan (dry-run)'); + info('Would fetch OpenRouter account activity with OPENROUTER_MANAGEMENT_KEY.'); + info(`Would replace the private normalized cache: ${cacheFile}`); + info(dim('No network request or file write was performed.')); + } + return 0; + } + try { + const value = await refresh({ cacheFile }); + if (flags.json) { + console.log(JSON.stringify({ cacheFile, openrouter: value }, null, 2)); + return 0; + } + const s = summary(value); + ok(`OpenRouter activity cached: ${s.totals.requests} requests · ${s.models} model(s)`); + info(`${s.coverage.from ?? 'no activity'} → ${s.coverage.through ?? 'no activity'} ` + + dim('· completed UTC days only')); + info(dim('cache contains no key, endpoint id, user id, session id, project, or prompt data')); + return 0; + } catch (error) { + if (flags.json) { + console.log(JSON.stringify({ cacheFile, error: String(error?.message ?? error) }, null, 2)); + } else { + warn(String(error?.message ?? error)); + } + return 1; + } + } + + warn('usage: ak usage status | ak usage refresh openrouter'); + return 2; +} diff --git a/src/lib/dashboard-server.mjs b/src/lib/dashboard-server.mjs index fdc28e7..afbc837 100644 --- a/src/lib/dashboard-server.mjs +++ b/src/lib/dashboard-server.mjs @@ -222,6 +222,11 @@ function lazyUsage() { readIndex: async (opts) => (await load()).readIndex(opts), readSession: async (id) => (await load()).readSession(id), maskSecrets: async (s) => (await load()).maskSecrets(s), + // Provider-account analytics is an independent offline cache read. It is + // deliberately not part of usage-index's transcript aggregation. + readProviderAnalytics: async () => ({ + openrouter: (await import('./usage-openrouter.mjs')).readOpenRouterActivity(), + }), // resolved at call time so a module that ships masking later still fails // closed rather than silently serving raw text. masker: async () => (await load()).maskSecrets, @@ -771,10 +776,19 @@ export function startDashboard({ // all" control still knows the true count and calls /api/sessions for the rest. if (url === '/api/usage') { try { - const agg = await usageApi.readIndex({ days: clampDays(query.get('days')) }); + const [agg, providerAnalytics] = await Promise.all([ + usageApi.readIndex({ days: clampDays(query.get('days')) }), + typeof usageApi.readProviderAnalytics === 'function' + ? Promise.resolve(usageApi.readProviderAnalytics()) + .catch(() => ({ openrouter: null })) + : Promise.resolve({ openrouter: null }), + ]); const { sessions: _sessions, projectTree, ...rollups } = agg || {}; sendJson(res, 200, { ...rollups, + // Account-level metadata has no session/host correlation key. Keep + // it visibly separate instead of laundering it into local totals. + providerAnalytics, projectTree: (projectTree || []).map((n) => ({ ...n, rowsTotal: Array.isArray(n.rows) ? n.rows.length : 0, diff --git a/src/lib/dashboard/client.mjs b/src/lib/dashboard/client.mjs index 597eac4..2e56018 100644 --- a/src/lib/dashboard/client.mjs +++ b/src/lib/dashboard/client.mjs @@ -689,6 +689,35 @@ export const JS = ` var exc=fld(t,"exceptions"); document.getElementById("u-models-note").textContent=exc?(fmtNum(exc)+" dropped/errored turn"+(exc===1?"":"s")+" excluded"):""; + // Account analytics is explicitly fetched and cached by ak usage. + // OpenRouter does not provide session/host/project correlation here, so + // these numbers remain a separate block and never alter t/byHost/byModel. + var ora=d.providerAnalytics&&d.providerAnalytics.openrouter; + if(!ora){ + document.getElementById("u-openrouter-note").textContent="not refreshed · offline"; + document.getElementById("u-openrouter").innerHTML= + '
No OpenRouter account cache. Run ak usage refresh openrouter explicitly.
'; + }else{ + var ot=ora.totals||{}, cov=ora.coverage||{}, oms=ora.byModel||[]; + document.getElementById("u-openrouter-note").textContent= + "cached "+ago(Math.max(0,Math.round((Date.now()-Date.parse(ora.fetchedAt))/1000)))+" · " + +(cov.from||"no activity")+" → "+(cov.through||"no activity") + +" · never merged into transcript totals"; + var oMax=oms.reduce(function(m,x){return Math.max(m,fld(x,"requests"));},0); + var cards='
' + +'
OpenRouter requests
'+esc(fmtNum(ot.requests))+'
' + +'
'+esc(fmtTok(fld(ot,"promptTokens")+fld(ot,"completionTokens")))+" tokens · 30 completed UTC days
" + +'
OpenRouter credits spent
'+esc(fmtUsd(ot.usage))+'
' + +'
'+esc(fmtUsd(ot.byokUsageInference))+" BYOK inference estimate · account-level
"; + var modelRows=oms.length?oms.map(function(m){ + var req=fld(m,"requests"); + return bar(esc(m.model),fmtNum(req)+" req",fmtTok(fld(m,"promptTokens")+fld(m,"completionTokens")) + +" tok · "+fmtUsd(fld(m,"usage"))+" OpenRouter credits · " + +fmtUsd(fld(m,"byokUsageInference"))+" BYOK estimate",pct(req,oMax),false); + }).join(""):'
The cache contains no completed activity.
'; + document.getElementById("u-openrouter").innerHTML=cards+'
'+modelRows+"
"; + } + var projects=entries(d.byProject), pMax=projects.length?projects[0].cost:0; var shown=projects.slice(0,8); document.getElementById("u-projects-note").textContent= diff --git a/src/lib/dashboard/page.mjs b/src/lib/dashboard/page.mjs index ece2bc8..d42f92e 100644 --- a/src/lib/dashboard/page.mjs +++ b/src/lib/dashboard/page.mjs @@ -192,6 +192,11 @@ export function renderPage({ name, version }) {
+
+

provider account analytics

+ offline cache · separate from transcript totals
+
+

what you worked on

classified from titles, skills & tool mix · click to filter
diff --git a/src/lib/dashboard/styles.mjs b/src/lib/dashboard/styles.mjs index 7ab850f..1518655 100644 --- a/src/lib/dashboard/styles.mjs +++ b/src/lib/dashboard/styles.mjs @@ -450,6 +450,7 @@ body.gated .band,body.gated .tabbar,body.gated main{display:none} .pcard .pv{font-size:21px; font-weight:700; letter-spacing:-.02em} .pcard .pl{font-size:11.5px; color:var(--ink-dim); margin-top:4px} .pcard.idle{opacity:.55} +.provider-analytics-models{margin-top:14px} .tokbar{display:flex; height:9px; border-radius:5px; overflow:hidden; margin-top:11px} .tokbar i{display:block; height:100%} .legend{display:flex; gap:14px; flex-wrap:wrap; margin-top:9px; font-size:11px; color:var(--ink-dim)} diff --git a/src/lib/usage-openrouter.mjs b/src/lib/usage-openrouter.mjs new file mode 100644 index 0000000..9cb0426 --- /dev/null +++ b/src/lib/usage-openrouter.mjs @@ -0,0 +1,288 @@ +// OpenRouter account analytics (issue #59). +// +// Network access is EXPLICIT: only refreshOpenRouterActivity() calls the +// management API. Every dashboard/status read uses the private local cache. +// The upstream response is grouped by endpoint, but endpoint ids are neither +// useful nor necessary for the scorecard, so normalization aggregates them +// away. The resulting account-level evidence must never be joined to local +// sessions: the API supplies no session, host, project, or task correlation. +import fs from 'node:fs'; +import path from 'node:path'; +import { configDir } from './paths.mjs'; + +export const OPENROUTER_ACTIVITY_URL = 'https://openrouter.ai/api/v1/activity'; +export const OPENROUTER_ACTIVITY_SCHEMA = 1; +export const OPENROUTER_ACTIVITY_DAYS = 30; +export const OPENROUTER_ACTIVITY_MAX_BYTES = 4 * 1024 * 1024; +export const openRouterActivityFile = () => path.join(configDir(), 'openrouter-activity.json'); + +const DATE_RE = /^\d{4}-\d{2}-\d{2}$/; +const METRICS = [ + 'requests', 'promptTokens', 'completionTokens', 'reasoningTokens', + 'usage', 'byokUsageInference', +]; + +function requiredNumber(row, key, index, { integer = false } = {}) { + const value = row[key]; + if (typeof value !== 'number' || !Number.isFinite(value) || value < 0 + || (integer && !Number.isInteger(value))) { + throw new TypeError(`OpenRouter activity row ${index} has invalid ${key}`); + } + return value; +} + +function requiredText(row, key, index, max = 256) { + const value = row[key]; + if (typeof value !== 'string' || value.length === 0 || value.length > max) { + throw new TypeError(`OpenRouter activity row ${index} has invalid ${key}`); + } + return value; +} + +function blankMetrics() { + return { + requests: 0, + promptTokens: 0, + completionTokens: 0, + reasoningTokens: 0, + usage: 0, + byokUsageInference: 0, + }; +} + +function addMetrics(target, row) { + for (const key of METRICS) target[key] += row[key]; + return target; +} + +function rounded(metrics) { + return { + ...metrics, + usage: Math.round(metrics.usage * 1e9) / 1e9, + byokUsageInference: Math.round(metrics.byokUsageInference * 1e9) / 1e9, + }; +} + +function normalizeRow(row, index) { + if (!row || typeof row !== 'object' || Array.isArray(row)) { + throw new TypeError(`OpenRouter activity row ${index} must be an object`); + } + const date = requiredText(row, 'date', index, 10); + const parsedDate = new Date(`${date}T00:00:00.000Z`); + if (!DATE_RE.test(date) || Number.isNaN(parsedDate.valueOf()) + || parsedDate.toISOString().slice(0, 10) !== date) { + throw new TypeError(`OpenRouter activity row ${index} has invalid date`); + } + const model = requiredText(row, 'model', index); + // Validate the endpoint id because it is required by the supported schema, + // then intentionally discard it before the cache shape is constructed. + requiredText(row, 'endpoint_id', index, 512); + return { + date, + model, + modelPermaslug: requiredText(row, 'model_permaslug', index), + providerName: requiredText(row, 'provider_name', index), + requests: requiredNumber(row, 'requests', index, { integer: true }), + promptTokens: requiredNumber(row, 'prompt_tokens', index, { integer: true }), + completionTokens: requiredNumber(row, 'completion_tokens', index, { integer: true }), + reasoningTokens: requiredNumber(row, 'reasoning_tokens', index, { integer: true }), + usage: requiredNumber(row, 'usage', index), + byokUsageInference: requiredNumber(row, 'byok_usage_inference', index), + }; +} + +function groupedRows(rows, keyOf, describe) { + const groups = new Map(); + for (const row of rows) { + const key = keyOf(row); + let group = groups.get(key); + if (!group) { + group = { ...describe(row), ...blankMetrics() }; + groups.set(key, group); + } + addMetrics(group, row); + } + return [...groups.values()] + .map((group) => ({ ...group, ...rounded(group) })) + .sort((a, b) => ( + b.requests - a.requests + || b.usage - a.usage + || b.byokUsageInference - a.byokUsageInference + || String(a.model ?? a.providerName).localeCompare(String(b.model ?? b.providerName)) + )); +} + +function completedUtcWindow(now) { + const through = new Date(now); + through.setUTCHours(0, 0, 0, 0); + through.setUTCDate(through.getUTCDate() - 1); + const from = new Date(through); + from.setUTCDate(from.getUTCDate() - (OPENROUTER_ACTIVITY_DAYS - 1)); + return { + completedUtcDays: OPENROUTER_ACTIVITY_DAYS, + from: from.toISOString().slice(0, 10), + through: through.toISOString().slice(0, 10), + }; +} + +/** + * Convert the management API response into the credential-free cache shape. + * Endpoint ids and any unknown upstream fields are deliberately discarded. + */ +export function normalizeOpenRouterActivity(raw, { now = Date.now() } = {}) { + if (!raw || typeof raw !== 'object' || !Array.isArray(raw.data)) { + throw new TypeError('OpenRouter activity response must contain data[]'); + } + const rows = raw.data.map((row, index) => normalizeRow(row, index)); + + // Multiple upstream endpoint rows can describe the same date/model/provider. + // Collapse them so the cache cannot retain endpoint identifiers by accident. + const daily = groupedRows( + rows, + (row) => JSON.stringify([row.date, row.model, row.modelPermaslug, row.providerName]), + (row) => ({ + date: row.date, + model: row.model, + modelPermaslug: row.modelPermaslug, + providerName: row.providerName, + }), + ).sort((a, b) => a.date.localeCompare(b.date) || a.model.localeCompare(b.model)); + + const totals = rounded(daily.reduce((sum, row) => addMetrics(sum, row), blankMetrics())); + const dates = [...new Set(daily.map((row) => row.date))].sort(); + return { + schemaVersion: OPENROUTER_ACTIVITY_SCHEMA, + provider: 'openrouter', + source: 'management-api/activity', + fetchedAt: new Date(now).toISOString(), + coverage: completedUtcWindow(now), + activitySpan: { + from: dates[0] ?? null, + through: dates.at(-1) ?? null, + }, + totals, + byModel: groupedRows( + daily, + (row) => JSON.stringify([row.model, row.modelPermaslug]), + (row) => ({ model: row.model, modelPermaslug: row.modelPermaslug }), + ), + byProvider: groupedRows( + daily, + (row) => row.providerName, + (row) => ({ providerName: row.providerName }), + ), + rows: daily, + }; +} + +function cacheShape(value) { + return value + && value.schemaVersion === OPENROUTER_ACTIVITY_SCHEMA + && value.provider === 'openrouter' + && typeof value.fetchedAt === 'string' + && value.totals && typeof value.totals === 'object' + && Array.isArray(value.byModel) + && Array.isArray(value.byProvider) + && Array.isArray(value.rows); +} + +/** Pure offline read. Missing, corrupt, or unknown-schema cache → null. */ +export function readOpenRouterActivity({ cacheFile = openRouterActivityFile() } = {}) { + try { + const value = JSON.parse(fs.readFileSync(cacheFile, 'utf8')); + return cacheShape(value) ? value : null; + } catch { + return null; + } +} + +function writeCache(cacheFile, value) { + fs.mkdirSync(path.dirname(cacheFile), { recursive: true }); + const tmp = `${cacheFile}.${process.pid}.tmp`; + try { + fs.writeFileSync(tmp, `${JSON.stringify(value, null, 2)}\n`, { mode: 0o600 }); + fs.renameSync(tmp, cacheFile); + try { fs.chmodSync(cacheFile, 0o600); } catch { /* best effort on Windows/exotic fs */ } + } catch (error) { + try { fs.rmSync(tmp, { force: true }); } catch { /* best effort */ } + throw error; + } +} + +async function boundedResponseText(response, maxBytes) { + const declared = Number(response.headers?.get?.('content-length')); + if (Number.isFinite(declared) && declared > maxBytes) { + throw new Error(`OpenRouter activity response exceeds ${maxBytes} bytes`); + } + if (!response.body || typeof response.body.getReader !== 'function') { + const body = await response.text(); + if (Buffer.byteLength(body) > maxBytes) { + throw new Error(`OpenRouter activity response exceeds ${maxBytes} bytes`); + } + return body; + } + + const reader = response.body.getReader(); + const chunks = []; + let total = 0; + while (true) { + const { done, value } = await reader.read(); + if (done) break; + total += value.byteLength; + if (total > maxBytes) { + try { await reader.cancel(); } catch { /* already closed */ } + throw new Error(`OpenRouter activity response exceeds ${maxBytes} bytes`); + } + chunks.push(value); + } + const joined = new Uint8Array(total); + let offset = 0; + for (const chunk of chunks) { + joined.set(chunk, offset); + offset += chunk.byteLength; + } + return new TextDecoder().decode(joined); +} + +/** + * Explicit network refresh. Requires a management key—not an inference key— + * and persists only the normalized credential-free cache. + */ +export async function refreshOpenRouterActivity({ + key = process.env.OPENROUTER_MANAGEMENT_KEY, + cacheFile = openRouterActivityFile(), + fetchImpl = globalThis.fetch, + timeoutMs = 15_000, + maxBytes = OPENROUTER_ACTIVITY_MAX_BYTES, + now = Date.now(), +} = {}) { + if (!key) { + throw new Error('OPENROUTER_MANAGEMENT_KEY is required (an inference API key is not sufficient)'); + } + if (typeof fetchImpl !== 'function') throw new TypeError('fetch is unavailable'); + + const controller = new AbortController(); + const timer = setTimeout(() => controller.abort(), timeoutMs); + let response; + try { + response = await fetchImpl(OPENROUTER_ACTIVITY_URL, { + method: 'GET', + headers: { + Accept: 'application/json', + Authorization: `Bearer ${key}`, + }, + signal: controller.signal, + }); + if (!response?.ok) { + throw new Error(`OpenRouter activity refresh failed (HTTP ${response?.status ?? 'unknown'})`); + } + const body = await boundedResponseText(response, maxBytes); + let raw; + try { raw = JSON.parse(body); } catch { throw new Error('OpenRouter activity response was not valid JSON'); } + const value = normalizeOpenRouterActivity(raw, { now }); + writeCache(cacheFile, value); + return value; + } finally { + clearTimeout(timer); + } +} diff --git a/tests/dashboard.test.cjs b/tests/dashboard.test.cjs index 94fe808..86988ca 100644 --- a/tests/dashboard.test.cjs +++ b/tests/dashboard.test.cjs @@ -465,6 +465,26 @@ async function main() { ], insights: [{ id: 'context-tax', kind: 'coach', severity: 'warn', title: 't', finding: 'f', evidence: 'e', action: 'a', command: null, impact: 3.5 }], }; + const PROVIDER_ANALYTICS = { + openrouter: { + schemaVersion: 1, + provider: 'openrouter', + source: 'management-api/activity', + fetchedAt: '2026-07-30T00:00:00.000Z', + coverage: { completedUtcDays: 30, from: '2026-07-01', through: '2026-07-29' }, + totals: { + requests: 3, promptTokens: 100, completionTokens: 50, + reasoningTokens: 10, usage: 0.25, byokUsageInference: 0.05, + }, + byModel: [{ + model: 'z-ai/glm-5.2', modelPermaslug: 'z-ai/glm-5.2', + requests: 3, promptTokens: 100, completionTokens: 50, + reasoningTokens: 10, usage: 0.25, byokUsageInference: 0.05, + }], + byProvider: [], + rows: [], + }, + }; function spyUsage(over = {}) { const calls = { readIndex: [], readSession: [] }; @@ -472,6 +492,7 @@ async function main() { calls, api: { readIndex: async (opts) => { calls.readIndex.push(opts); return JSON.parse(JSON.stringify(AGG)); }, + readProviderAnalytics: async () => JSON.parse(JSON.stringify(PROVIDER_ANALYTICS)), readSession: async (id) => { calls.readSession.push(id); return { meta: { id, title: 'one', project: 'demo' }, turns: [{ role: 'user', text: 'token sk-live-DEADBEEF01234 pasted' }] }; @@ -494,6 +515,10 @@ async function main() { const j = JSON.parse(r.body); assert(!('sessions' in j), 'sessions[] must be stripped — that is what /api/sessions is for'); assert(j.totals && j.totals.cost === 12.5, 'totals must survive'); + assert(j.providerAnalytics.openrouter.totals.requests === 3, + 'provider analytics must travel in its own top-level block'); + assert(j.totals.sessions === 2 && j.totals.tokens === 1250, + 'provider analytics must not alter transcript totals'); assert(j.projectTree && j.projectTree.length === 1, 'projectTree must survive'); assert(Array.isArray(j.insights) && j.insights.length === 1, 'insights must survive'); assert(spy.calls.readIndex.some((o) => o && o.days === 7), 'days must reach readIndex, got ' + JSON.stringify(spy.calls.readIndex)); @@ -621,6 +646,13 @@ async function main() { contains(r.body, 'id="v-' + v + '"'); contains(r.body, 'data-view="' + v + '"'); } + contains(r.body, 'id="u-openrouter"'); + contains(r.body, 'provider account analytics'); + contains(r.body, 'never merged into transcript totals'); + contains(r.body, 'OpenRouter credits'); + contains(r.body, 'BYOK estimate'); + assert(!r.body.includes('fld(x,"usage")+fld(x,"byokUsageInference")'), + 'OpenRouter-credit usage and BYOK external cost must not be silently combined'); }); await test('Usage rendering treats host and inference provider as independent axes', async () => { diff --git a/tests/kit/cli-help.test.mjs b/tests/kit/cli-help.test.mjs index 6bf47e7..f9e4903 100644 --- a/tests/kit/cli-help.test.mjs +++ b/tests/kit/cli-help.test.mjs @@ -30,7 +30,7 @@ test('mutating commands intercept both --help and -h before running', () => { }); test('every command exposes an Examples section in its help', () => { - for (const cmd of [['setup'], ['status'], ['sync'], ['run'], ['dual'], ['dashboard'], ['uninstall'], + for (const cmd of [['setup'], ['status'], ['sync'], ['usage'], ['run'], ['dual'], ['dashboard'], ['uninstall'], ['host'], ['provider'], ['x', 'mcp'], ['x', 'host'], ['x', 'provider'], ['x', 'verify'], ['x', 'reference'], ['x', 'daemon-gc']]) { const r = ak(...cmd, '--help'); diff --git a/tests/kit/usage-cli.test.mjs b/tests/kit/usage-cli.test.mjs new file mode 100644 index 0000000..dc108f7 --- /dev/null +++ b/tests/kit/usage-cli.test.mjs @@ -0,0 +1,94 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { spawnSync } from 'node:child_process'; +import { fileURLToPath } from 'node:url'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; + +const BIN = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '../../bin/agentic-kit.mjs'); + +function sandbox() { + const home = fs.mkdtempSync(path.join(os.tmpdir(), 'ak-usage-cli-')); + const cfg = path.join(home, '.config'); + const bin = path.join(home, 'bin'); + const sentinel = path.join(home, 'npm-was-called'); + fs.mkdirSync(bin, { recursive: true }); + fs.writeFileSync(path.join(bin, 'npm'), `#!/bin/sh\nprintf called > "${sentinel}"\nexit 99\n`, { mode: 0o755 }); + fs.writeFileSync(path.join(bin, 'npm.cmd'), `@echo called>"${sentinel}"\r\nexit /b 99\r\n`); + return { home, cfg, bin, sentinel }; +} + +function ak(args, sb, extra = {}) { + return spawnSync(process.execPath, [BIN, ...args], { + encoding: 'utf8', + env: { + ...process.env, + NO_COLOR: '1', + HOME: sb.home, + USERPROFILE: sb.home, + XDG_CONFIG_HOME: sb.cfg, + APPDATA: sb.cfg, + PATH: `${sb.bin}${path.delimiter}${process.env.PATH ?? ''}`, + OPENROUTER_MANAGEMENT_KEY: '', + ...extra, + }, + }); +} + +test('ak usage status is an offline cache read with no generic npm drift probe', () => { + const sb = sandbox(); + const result = ak(['usage', 'status'], sb); + assert.equal(result.status, 0, result.stderr); + assert.match(result.stdout, /no local activity cache/i); + assert.equal(fs.existsSync(sb.sentinel), false, 'offline status must never execute npm'); + fs.rmSync(sb.home, { recursive: true, force: true }); +}); + +test('ak usage refresh openrouter requires the dedicated management key', () => { + const sb = sandbox(); + const result = ak(['usage', 'refresh', 'openrouter'], sb, { + OPENROUTER_API_KEY: 'inference-only-key', + }); + assert.equal(result.status, 1); + assert.match(result.stdout, /OPENROUTER_MANAGEMENT_KEY is required/); + assert.equal(result.stdout.includes('inference-only-key'), false); + assert.equal(fs.existsSync(sb.sentinel), false, 'failure must not fall through to npm drift'); + fs.rmSync(sb.home, { recursive: true, force: true }); +}); + +test('ak usage refresh openrouter --dry-run performs no network or writes', () => { + const sb = sandbox(); + const result = ak(['usage', 'refresh', 'openrouter', '--dry-run', '--json'], sb, { + OPENROUTER_MANAGEMENT_KEY: 'must-not-be-used', + }); + assert.equal(result.status, 0, result.stderr); + const value = JSON.parse(result.stdout); + assert.deepEqual( + { + dryRun: value.dryRun, + action: value.action, + provider: value.provider, + network: value.network, + writes: value.writes, + }, + { + dryRun: true, + action: 'refresh', + provider: 'openrouter', + network: false, + writes: false, + }, + ); + assert.equal(fs.existsSync(path.join(sb.cfg, 'agentic-kit', 'openrouter-activity.json')), false); + assert.equal(fs.existsSync(sb.sentinel), false); + fs.rmSync(sb.home, { recursive: true, force: true }); +}); + +test('ak usage rejects unsupported providers and actions', () => { + const sb = sandbox(); + const result = ak(['usage', 'refresh', 'unknown'], sb); + assert.equal(result.status, 2); + assert.match(result.stdout, /usage: ak usage status/); + fs.rmSync(sb.home, { recursive: true, force: true }); +}); diff --git a/tests/kit/usage-openrouter.test.mjs b/tests/kit/usage-openrouter.test.mjs new file mode 100644 index 0000000..1ecc7ce --- /dev/null +++ b/tests/kit/usage-openrouter.test.mjs @@ -0,0 +1,212 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import os from 'node:os'; +import path from 'node:path'; +import { + OPENROUTER_ACTIVITY_URL, + normalizeOpenRouterActivity, + readOpenRouterActivity, + refreshOpenRouterActivity, +} from '../../src/lib/usage-openrouter.mjs'; + +const tmp = () => fs.mkdtempSync(path.join(os.tmpdir(), 'ak-openrouter-usage-')); +const RAW = { + data: [ + { + byok_usage_inference: 0.012, + completion_tokens: 125, + date: '2026-07-28', + endpoint_id: 'endpoint-secret-a', + model: 'openai/gpt-5.6', + model_permaslug: 'openai/gpt-5.6-20260701', + prompt_tokens: 50, + provider_name: 'OpenAI', + reasoning_tokens: 25, + requests: 5, + usage: 0.015, + }, + { + byok_usage_inference: 0.003, + completion_tokens: 75, + date: '2026-07-28', + endpoint_id: 'endpoint-secret-b', + model: 'openai/gpt-5.6', + model_permaslug: 'openai/gpt-5.6-20260701', + prompt_tokens: 25, + provider_name: 'OpenAI', + reasoning_tokens: 10, + requests: 2, + usage: 0.009, + }, + { + byok_usage_inference: 0, + completion_tokens: 30, + date: '2026-07-29', + endpoint_id: 'endpoint-secret-c', + model: 'z-ai/glm-5.2', + model_permaslug: 'z-ai/glm-5.2', + prompt_tokens: 20, + provider_name: 'Z.AI', + reasoning_tokens: 0, + requests: 1, + usage: 0.004, + }, + ], +}; + +test('normalization aggregates endpoint rows and discards every correlation identifier', () => { + const out = normalizeOpenRouterActivity(RAW, { now: Date.parse('2026-07-30T00:00:00Z') }); + assert.equal(out.provider, 'openrouter'); + assert.equal(out.fetchedAt, '2026-07-30T00:00:00.000Z'); + assert.deepEqual(out.coverage, { + completedUtcDays: 30, + from: '2026-06-30', + through: '2026-07-29', + }); + assert.deepEqual(out.activitySpan, { + from: '2026-07-28', + through: '2026-07-29', + }); + assert.equal(out.rows.length, 2, 'two endpoint rows for one model/day collapse'); + assert.equal(out.rows[0].requests, 7); + assert.equal(out.rows[0].promptTokens, 75); + assert.equal(out.rows[0].completionTokens, 200); + assert.equal(out.totals.requests, 8); + assert.equal(out.totals.promptTokens, 95); + assert.equal(out.totals.completionTokens, 230); + assert.equal(out.byModel.length, 2); + assert.equal(out.byProvider.length, 2); + + const wire = JSON.stringify(out); + for (const forbidden of ['endpoint_id', 'endpoint-secret', 'api_key_hash', 'user_id', 'session']) { + assert.equal(wire.includes(forbidden), false, `${forbidden} must not enter the cache`); + } +}); + +test('normalization accepts an honest empty window and rejects any malformed row', () => { + const empty = normalizeOpenRouterActivity({ data: [] }, { now: 0 }); + assert.equal(empty.rows.length, 0); + assert.equal(empty.totals.requests, 0); + assert.throws( + () => normalizeOpenRouterActivity({ data: [{ endpoint_id: 'only-an-id' }] }), + /invalid date/, + ); + assert.throws(() => normalizeOpenRouterActivity({ + data: [RAW.data[0], { ...RAW.data[1], provider_name: undefined }], + }), /row 1 has invalid provider_name/); + assert.throws(() => normalizeOpenRouterActivity({ + data: [{ ...RAW.data[0], requests: -1 }], + }), /row 0 has invalid requests/); + assert.throws(() => normalizeOpenRouterActivity({ + data: [{ ...RAW.data[0], completion_tokens: 1.5 }], + }), /row 0 has invalid completion_tokens/); + assert.throws(() => normalizeOpenRouterActivity({}), /data\[\]/); +}); + +test('refresh uses only the management key, writes a private atomic cache, and never stores the key', async () => { + const dir = tmp(); + const cacheFile = path.join(dir, 'openrouter-activity.json'); + const key = 'management-secret-must-not-land'; + let request; + const fetchImpl = async (url, init) => { + request = { url, init }; + return new Response(JSON.stringify(RAW), { + status: 200, + headers: { 'content-type': 'application/json' }, + }); + }; + + const out = await refreshOpenRouterActivity({ + key, + cacheFile, + fetchImpl, + now: Date.parse('2026-07-30T01:00:00Z'), + }); + assert.equal(request.url, OPENROUTER_ACTIVITY_URL); + assert.equal(request.init.method, 'GET'); + assert.equal(request.init.headers.Authorization, `Bearer ${key}`); + assert.ok(request.init.signal instanceof AbortSignal); + assert.equal(out.totals.requests, 8); + + const wire = fs.readFileSync(cacheFile, 'utf8'); + assert.equal(wire.includes(key), false); + assert.equal(wire.includes('endpoint-secret'), false); + if (process.platform !== 'win32') assert.equal(fs.statSync(cacheFile).mode & 0o777, 0o600); + assert.deepEqual(readOpenRouterActivity({ cacheFile }), out); + fs.rmSync(dir, { recursive: true, force: true }); +}); + +test('an inference key is never an implicit substitute for a management key', async () => { + await assert.rejects( + refreshOpenRouterActivity({ + key: null, + fetchImpl: async () => { throw new Error('must not fetch'); }, + }), + /OPENROUTER_MANAGEMENT_KEY is required/, + ); +}); + +test('HTTP/schema/size failures preserve the last good cache', async () => { + const dir = tmp(); + const cacheFile = path.join(dir, 'openrouter-activity.json'); + fs.writeFileSync(cacheFile, '{"sentinel":"old"}'); + + await assert.rejects( + refreshOpenRouterActivity({ + key: 'management-key', + cacheFile, + fetchImpl: async () => new Response('nope', { status: 403 }), + }), + /HTTP 403/, + ); + assert.equal(fs.readFileSync(cacheFile, 'utf8'), '{"sentinel":"old"}'); + + await assert.rejects( + refreshOpenRouterActivity({ + key: 'management-key', + cacheFile, + fetchImpl: async () => new Response(JSON.stringify({ + data: [RAW.data[0], { ...RAW.data[1], reasoning_tokens: null }], + }), { status: 200 }), + }), + /invalid reasoning_tokens/, + ); + assert.equal(fs.readFileSync(cacheFile, 'utf8'), '{"sentinel":"old"}'); + + await assert.rejects( + refreshOpenRouterActivity({ + key: 'management-key', + cacheFile, + maxBytes: 10, + fetchImpl: async () => ({ + ok: true, + status: 200, + headers: { get: () => '100' }, + text: async () => JSON.stringify(RAW), + }), + }), + /exceeds 10 bytes/, + ); + assert.equal(fs.readFileSync(cacheFile, 'utf8'), '{"sentinel":"old"}'); + fs.rmSync(dir, { recursive: true, force: true }); +}); + +test('offline reads fail closed on corrupt and unknown-schema caches', () => { + const dir = tmp(); + const cacheFile = path.join(dir, 'openrouter-activity.json'); + assert.equal(readOpenRouterActivity({ cacheFile }), null); + fs.writeFileSync(cacheFile, '{bad json'); + assert.equal(readOpenRouterActivity({ cacheFile }), null); + fs.writeFileSync(cacheFile, JSON.stringify({ + schemaVersion: 999, + provider: 'openrouter', + fetchedAt: '2026-07-30T00:00:00Z', + totals: {}, + byModel: [], + byProvider: [], + rows: [], + })); + assert.equal(readOpenRouterActivity({ cacheFile }), null); + fs.rmSync(dir, { recursive: true, force: true }); +});