ai(proforge): replace character-count token metrics with provider usage telemetry

[qnbs]

Context

Execution owner for finding #3 from the 2026-09-12 /claude-api prompt-audit, under umbrella #704 and backed by prep/evidence PR #727.

Prep/evidence PR: #727 — docs(proforge): plan real token accounting for the Claude path

Plan document: docs/PROFORGE-CLAUDE-TOKEN-ACCOUNTING-PLAN.md

Current main baseline at issue creation:

021b1bdc145f211cf4f4239db7ad26e44ca7fae2

The prep PR is planning evidence, not implementation authority. Re-fetch the current call graph and current provider usage contracts before implementation.

Related audit owners:

• ai(proforge): use native Claude structured outputs for schema-bound pipeline stages #729 + feat(proforge): plan Claude structured-outputs wiring #730 — Claude native structured outputs;
• ai(claude): right-size ProForge response budgets and implement real end-to-end streaming #731 + docs(proforge): plan Claude max_tokens/timeout right-sizing #728 — Claude output budgets / timeout / real streaming;
• ai: requalify current model catalog, defaults and end-to-end AI functionality across all providers #704 — provider/model capability and real-provider qualification authority.

---

Audit finding

ProForge currently labels raw response character counts as token consumption while the Claude path already receives provider usage metadata that is not threaded through the application result contract.

The audit identified examples such as:

services/aiProviderService.ts
  Anthropic final response is parsed
  response text is extracted
  response usage metadata is not returned to callers

services/proForge/pipelineAgents/structuralAgent.ts
services/proForge/pipelineAgents/publishingAgent.ts
services/proForge/pipelineAgents/baseAgent.ts
  tokensConsumed += response.length
  tokensUsed: response.text.length

Raw JavaScript string length is characters/code units, not tokens.

This makes metrics misleading and prevents #731's output-budget/timeout tuning from being driven by real provider evidence.

The audit proposed a two-stage remediation:

Phase A:
stop reporting raw character counts as tokens;
use the repository's existing estimateTokens() helper when real usage is unavailable.

Phase B:
thread actual provider-reported usage through the inference result contract;
use exact usage whenever the provider supplies it;
fall back to an explicitly-estimated value otherwise.

This issue owns both phases so the interim approximation cannot silently become the permanent telemetry architecture.

---

Why this matters

Incorrect token accounting affects more than a cosmetic metric.

It can distort:

ProForge pipeline analytics
capacity planning
output-budget tuning
latency/cost interpretation
comparison between runs
future cost UX
provider qualification evidence

A 10,000-character response is not a 10,000-token response. The ratio varies by language, punctuation, code/JSON density and tokenizer/model family.

The current metrics schema already exposes concepts such as:

totalAiCalls
totalTokensConsumed

so the application should either report a real token count or clearly identify an estimate.

---

Phase A — immediately stop mislabeling characters as tokens

1. Inventory every affected metric write

Search current source for:

response.length
text.length
tokensConsumed
tokensUsed
totalTokensConsumed
estimateTokens

Do not assume the audit's three cited locations remain exhaustive.

Classify each occurrence as:

CHARACTER_COUNT_INTENTIONALLY
TOKEN_ESTIMATE
PROVIDER_REPORTED_USAGE
MISLABELED_CHARACTER_COUNT
UNUSED/DEAD

2. Reuse the existing estimator

Where real provider usage is not yet available and a token-like metric is genuinely needed, reuse the repository's current canonical estimation helper rather than adding a second character/token heuristic.

The prep PR identified estimateTokens() in the existing prompt/RAG tooling as the likely authority; verify its current location and suitability before importing it into ProForge.

If its semantics are model-agnostic approximate counting, preserve that limitation explicitly.

3. Do not pretend an estimate is exact

If the data model/UI can represent provenance without disproportionate churn, distinguish:

exact provider usage
estimated usage
unknown usage

If current public types cannot change safely in the small Phase-A PR, document the estimate semantics clearly and ensure Phase B remains an explicit acceptance requirement of this issue.

4. Preserve analytics compatibility

Do not break persisted ProForge run data unnecessarily.

If totalTokensConsumed is persisted/exported, define compatibility semantics for historical values that may have been character counts.

Do not retroactively rewrite old run history with invented token values unless a deterministic migration is justified.

---

Phase B — provider usage becomes first-class inference metadata

Core invariant

When a provider supplies authoritative usage:

provider response usage
        ↓
provider adapter/service
        ↓
inference/gateway result
        ↓
ProForge agent/run accounting
        ↓
pipeline analytics

No layer should discard the usage object only to reconstruct a poorer approximation later.

---

1. Introduce a bounded provider-neutral usage contract

Extend the appropriate inference result boundary with a small normalized usage type.

Conceptually:

interface AIUsage {
  inputTokens?: number;
  outputTokens?: number;
  totalTokens?: number;
  source: 'provider' | 'estimated';
  provider?: string;
}

Exact names are not prescribed.

Do not over-generalize speculative fields for every provider.

At implementation time, re-check the usage metadata actually supplied by the currently supported Anthropic API and other adapters.

Provider-specific cache/thinking/service metadata may be preserved separately only if WorldScript has a concrete consumer; otherwise normalize the stable fields required by current product analytics.

2. Requalify the existing gateway result contract

services/ai/inferenceGateway.ts currently returns a GenerateResult containing text/model/provider/fallback metadata but no usage.

Evaluate this as the preferred renderer-neutral propagation boundary instead of introducing ProForge-only side channels.

A likely direction is:

GenerateResult
├── text
├── model
├── provider
├── isFallback
└── usage?

but derive the final shape from current architecture.

Do not put mutable global lastUsage state into aiProviderService; concurrent requests would make it race-prone and attribution-unsafe.

3. Anthropic non-stream response usage

The current Claude response already carries provider usage metadata.

Extract and normalize it before the response shape is collapsed to plain text.

Validate fields defensively:

finite
non-negative
integer where provider contract requires integer

Malformed usage must not corrupt generation success; treat usage as unavailable and retain the text response unless provider contract integrity requires otherwise.

4. Anthropic streaming usage

Coordinate with #731.

When genuine SSE streaming lands, usage may arrive across message lifecycle events rather than one final JSON response.

The normalized result must settle only after the final provider usage is known.

Do not double-count partial cumulative usage or individual stream deltas.

Streaming implementation and usage implementation must define one ownership boundary for final accounting.

5. Fallback-chain attribution

WorldScript can route/fallback across providers.

The final usage record must describe the provider/model that actually performed the generation, not merely the originally requested provider.

If one failed provider consumed billable tokens before fallback and the API exposes that usage, decide explicitly whether run analytics account for:

successful-result usage only
or
all attempted-provider usage

Do not silently mix the two.

At minimum, the semantics of totalTokensConsumed must be documented and tested.

6. Estimated fallback usage

For providers that do not expose usable token metadata through the current adapter:

provider usage absent
→ estimateTokens(text)
→ mark source/provenance as estimated where architecture permits

Do not use .length as the fallback.

Do not call a remote tokenizer merely to compute analytics.

7. Input-token accounting

The audit primarily found output characters mislabeled as tokens, but the provider supplies input/output usage separately.

Define whether ProForge's totalTokensConsumed means:

output tokens only
or
input + output tokens

and make that meaning consistent across:

agent stats
pipeline metrics
analytics reports
future cost estimation
qualification reports

Prefer explicit fields if the existing single total is ambiguous.

Do not change semantics silently while keeping the same field name.

---

Privacy / security

Usage telemetry must remain content-free.

Do not persist or log:

API keys
prompts
manuscript prose
raw model outputs
account identifiers
provider request IDs unless there is a concrete sanitized diagnostic need

Safe metrics may include:

provider
model
feature/stage
input token count
output token count
estimated/provider provenance
duration
success/error class
application SHA

Respect the local-first/privacy architecture.

This issue is not authorization to add remote analytics collection.

---

Testing

Phase-A estimator regressions

For every migrated call site:

• prove .length is no longer used as token accounting;
• prove the canonical estimator is invoked when exact usage is absent;
• test non-ASCII / multilingual strings where character/token ratios visibly differ;
• test empty output and very short output;
• preserve AI-call counts independently from token counts.

Provider usage parsing

Fixture-test:

valid input/output usage
zero usage
missing usage
malformed/negative/non-number usage
provider response with valid text but invalid usage metadata

Gateway propagation

Prove normalized usage survives:

provider adapter
→ generateText/generation result boundary
→ InferenceGateway
→ BaseAgent/ProForge stage
→ aggregate pipeline metrics

No test should require a real API key.

Fallback tests

Cover:

primary provider success
fallback provider success
provider usage unavailable → estimate
provider usage present → exact wins
no double counting across retries/fallback

Streaming tests

Once #731 is implemented, cover final usage settlement from streaming fixtures and prove chunk callbacks do not increment token metrics independently.

---

Interaction with pipeline analytics

Requalify:

pipelineMetricsSchema.totalTokensConsumed
pipelineAnalyticsReportSchema
stage/run summaries
any UI that labels values as tokens/cost

If historical values are semantically mixed, document the version boundary or provenance rather than implying cross-run comparability that does not exist.

Do not implement monetary cost display until model pricing authority and exact accounting semantics are robust enough; that can remain under #704 if not already owned.

---

Secondary audit notes intentionally NOT split into new issues

The source audit explicitly identified two low-confidence observations that do not warrant standalone tickets now:

1. BaseAgent retry feedback such as IMPORTANT — your previous attempt was rejected... carries genuine per-attempt failure context and should remain.
2. The rigid COHERENT: / INCOHERENT: self-reflection prefix is load-bearing because the caller performs exact string matching. Revisit only after ai(proforge): use native Claude structured outputs for schema-bound pipeline stages #729's structured-output architecture proves a smaller/better contract.

Do not create prompt-cleanup churn from these notes.

---

Non-goals

This issue does NOT own:

• output ceilings/timeouts/stream transport — ai(claude): right-size ProForge response budgets and implement real end-to-end streaming #731 / docs(proforge): plan Claude max_tokens/timeout right-sizing #728;
• structured-output schema plumbing — ai(proforge): use native Claude structured outputs for schema-bound pipeline stages #729 / feat(proforge): plan Claude structured-outputs wiring #730;
• full provider/model refresh — ai: requalify current model catalog, defaults and end-to-end AI functionality across all providers #704;
• external telemetry service or user tracking;
• retroactive guessing of exact historical token counts;
• a universal tokenizer implementation for every model;
• pricing/billing UI unless separately admitted under ai: requalify current model catalog, defaults and end-to-end AI functionality across all providers #704;
• logging prompts or responses for measurement;
• changing AI policy/routing semantics.

---

Acceptance criteria

• Prep PR docs(proforge): plan real token accounting for the Claude path #727 and its plan are incorporated/requalified rather than re-audited from scratch.
• All raw character-count-as-token call sites are inventoried.
• No ProForge metric reports string.length as a token count.
• Canonical estimated-token logic is used wherever provider usage is unavailable.
• Estimate vs provider-exact provenance is explicit where practical and never falsely described as exact.
• Current Anthropic usage fields are re-verified from official authority at implementation time.
• Provider-reported usage is not discarded before the inference result boundary.
• GenerateResult or an equivalent renderer-neutral result carries normalized usage metadata.
• ProForge stage/run analytics consume exact provider usage when available.
• Input-vs-output-vs-total token semantics are explicitly defined.
• Fallback/retry accounting cannot double-count silently.
• Streaming usage integrates cleanly with ai(claude): right-size ProForge response budgets and implement real end-to-end streaming #731 and settles once.
• Local/provider paths without exact usage fall back to estimates, not character count.
• Historical persisted metrics are preserved or versioned without invented migration values.
• No prompt/manuscript/secret data is introduced into telemetry/logging.
• Deterministic tests cover usage parsing, propagation, fallback, estimation and aggregation.
• Final implementation evidence is linked back to ai: requalify current model catalog, defaults and end-to-end AI functionality across all providers #704 and docs(proforge): plan real token accounting for the Claude path #727.
• Exact-head CI, CodeQL, review/thread convergence and resulting-main verification complete before terminal closure.

---

Priority

P2, under P1 umbrella #704.

The current metrics are materially inaccurate and obstruct evidence-based tuning, but the audit did not demonstrate direct data corruption or a security/privacy bypass. The small estimator correction is low-risk; the provider-usage plumbing should be implemented deliberately with #731 so the result contract remains coherent.