Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/TRANSCRIPTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ rewritten; rule 3 of the module header, `usage-index.mjs:22-29`):
| Host | Store | Discovered by |
|---|---|---|
| Claude Code | `~/.claude/projects/<encoded-project-dir>/<sessionId>.jsonl` | `listClaude` (`usage-index.mjs:684`) — exactly one level of project directories |
| Codex CLI | `~/.codex/sessions/<yyyy>/<mm>/<dd>/rollout-<ts>-<uuid>.jsonl` | `listCodex` (`usage-index.mjs:663`) — the `yyyy/mm/dd` tree walk |
| Codex CLI | `~/.codex/sessions/<yyyy>/<mm>/<dd>/rollout-<ts>-<uuid>.jsonl` | `listCodex` (`usage-index.mjs:705`) — the `yyyy/mm/dd` tree walk |

Roots come from `defaultRoots()` (`usage-index.mjs:676`) and are injectable
for tests. A malformed line is skipped, never fatal (`jsonLines`,
Expand Down Expand Up @@ -96,7 +96,7 @@ The same parsers serve two very different callers, switched by `withTurns`:
| Path | Entry point | `withTurns` | Message bodies | Cached? |
|---|---|---|---|---|
| **Scan** — the aggregate index behind the Scorecard/Findings/Sessions views | `buildIndex` → `parseFile` (`usage-index.mjs:693`) | `false` | never held — holding them would balloon memory across 3,000+ files (`usage-index.mjs:437-440`) | yes: per-file derived records in `~/.config/agentic-kit/usage-index.json`, keyed `(path, mtime, size)`, invalidated wholesale by `SCHEMA_VERSION` (`usage-index.mjs:51`) |
| **Reader** — one transcript for the Transcript view | `readSession` (`usage-index.mjs:1173`) | `true` | full turn list built | **never** — every call re-reads and re-parses the one file |
| **Reader** — one transcript for the Transcript view | `readSession` (`usage-index.mjs:1310`) | `true` | full turn list built | **never** — every call re-reads and re-parses the one file |

![Figure: one parser, two read paths — the scan path (withTurns false) caches per-file records keyed by path, mtime and size; the reader path (withTurns true) builds full turns and is never cached](assets/transcript-read-paths.svg)

Expand Down Expand Up @@ -182,11 +182,11 @@ transcript content leaves the module, and every step is a gate:

1. **Id grammar before any filesystem access** — `VALID_ID`
(`/^[A-Za-z0-9._-]{1,128}$/`, `usage-index.mjs:71`) rejects traversal
shapes with `ERR_INVALID_SESSION_ID` (`usage-index.mjs:1215`).
2. **Locate by id** across both roots (`locate`, `usage-index.mjs:1222`),
shapes with `ERR_INVALID_SESSION_ID` (`usage-index.mjs:1260`).
2. **Locate by id** across both roots (`locate`, `usage-index.mjs:1267`),
consulting the scan cache when present but never requiring it —
`readSession` works with no prior `buildIndex`.
3. **Realpath containment** (`usage-index.mjs:1180-1194`) — the resolved file
3. **Realpath containment** (`usage-index.mjs:1335-1349`) — the resolved file
must live under a transcript root *after* `realpathSync` collapses
symlinks; a symlink planted inside a root pointing at `/etc/anything`
passes a lexical `startsWith` but fails this. Roots are realpath'd too so
Expand All @@ -211,7 +211,7 @@ Every turn body is passed through `maskSecrets` (`usage-index.mjs:184` — the
23 secret shapes) **server-side, before
serialization**, then length-capped at `MAX_TURN_CHARS` (40,000,
`usage-index.mjs:65`) with the marker appended
(`usage-index.mjs:1339-1348`). Two invariants:
(`usage-index.mjs:1404-1414`). Two invariants:

- **Presence is the signal.** `truncated`/`originalChars` are emitted only
when the slice fired, so a complete turn cannot be misread as abridged.
Expand Down
16 changes: 8 additions & 8 deletions docs/USAGE-SCORECARD-METRICS.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,9 +97,9 @@ responses = Σ over included sessions of session.responses
**Source:**

- Filter: a parsed record with zero assistant turns is dropped entirely — "no
assistant turn → not a session" (`usage-index.mjs:877`) — and a record whose
assistant turn → not a session" (`usage-index.mjs:892`) — and a record whose
last activity falls outside the requested window is dropped too
(`usage-index.mjs:878`).
(`usage-index.mjs:893`).
- `responses` accumulation: Claude increments per assistant message
(`usage-index.mjs:493`); Codex increments per `agent_message` event
(`usage-index.mjs:631`).
Expand Down Expand Up @@ -182,7 +182,7 @@ already in effect on the given day, comparing ISO date strings
lexicographically so no `Date` parsing is involved and the module stays
clock-free.

`aggregate()` passes each usage row's own `day` (`usage-index.mjs:877`), which
`aggregate()` passes each usage row's own `day` (`usage-index.mjs:892`), which
it already has because rows are keyed by `(day, model)`. **This is the whole
point:** tokens metered in August must still read as August's rate when the
panel is opened in December. Pricing by *today's* date instead would restate a
Expand Down Expand Up @@ -264,7 +264,7 @@ tokens = input + output + cacheRead + cacheWrite (summed across all rows in wi
```

**Source:** `t.tokens` from `totals`, accumulated per row at
`usage-index.mjs:824` (`rowTokens = row.input + row.output + row.cacheRead +
`usage-index.mjs:916` (`rowTokens = row.input + row.output + row.cacheRead +
row.cacheWrite`) and rolled into `totals.tokens` via `addTo`
(`usage-index.mjs:843-852`). Rendered with `fmtTok()`
(`dashboard/client.mjs`): `≥1e9` → `"X.XB"`, `≥1e6` → `"X.XM"`,
Expand Down Expand Up @@ -532,7 +532,7 @@ byModel[model].sessions = count of DISTINCT sessions whose s.models includes th
```

**Source:** cost/tokens/responses accumulate inside the usage-row loop
(`usage-index.mjs:814-831`); the `sessions` count is deliberately computed
(`usage-index.mjs:896-922`); the `sessions` count is deliberately computed
**separately**, once per session over its `s.models` array
(`usage-index.mjs:898-903`) rather than inside the cost loop, precisely
**so that a model can appear in `byModel` — with a nonzero session count —
Expand All @@ -544,7 +544,7 @@ excluded subagent-replay session still shows up as "used," at zero cost,
rather than vanishing.

`byModel[...].responses` is populated from `row.responses`
(`usage-index.mjs:829`), which in turn comes from the `responses` field
(`usage-index.mjs:921`), which in turn comes from the `responses` field
passed into `addUsage()` at the call site — `1` per Claude assistant turn
(`usage-index.mjs:484-490`), or `rec.responses` (the session's whole response
count) once per Codex session, passed at the single point Codex calls
Expand Down Expand Up @@ -880,7 +880,7 @@ Codex ≥0.140 maintains its own SQLite thread ledger (`~/.codex/state_N.sqlite`
— the `N` is a migration generation, so `codexStateDb` (`codex-state.mjs:30`)
globs and takes the newest). `readCodexState` (`:49`) reads per-thread
`thread_source` (`user` vs `subagent`) plus `thread_spawn_edges`, and
`applyCodexLedger` (`usage-index.mjs:1179`) overlays that onto parsed
`applyCodexLedger` (`usage-index.mjs:1218`) overlays that onto parsed
sessions: a ledger-identified subagent has its token usage stripped — its
rollout replays the parent's entire token history (ccusage/ccusage#950
measured up to 91× inflation) — while the session record stays visible. The
Expand Down Expand Up @@ -933,7 +933,7 @@ commit `540be18` on this branch.
Claude's parser passes `responses: 1` per assistant turn
(`usage-index.mjs:489`, as it existed before this fix), but Codex's call
passed no such field at all. Because `byModel[model].responses` is summed
directly from each usage row's `responses` field (`usage-index.mjs:829`,
directly from each usage row's `responses` field (`usage-index.mjs:921`,
`m.responses += row.responses`), **every** Codex model in §10's "Models in
Play" list displayed `0 resp` regardless of real token/cost volume or actual
`agent_message` count. **Fix:** `parseCodex` now passes `responses:
Expand Down
85 changes: 71 additions & 14 deletions src/lib/usage-index.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,7 @@ import fs from 'node:fs';
import path from 'node:path';
import { configDir, claudeDir, codexDir } from './paths.mjs';
import { readCodexState } from './codex-state.mjs';
import { defaultOpencodeDbPath, listSessions as listOpencodeSessions, parseSession as parseOpencodeSession, sessionExists as opencodeSessionExists } from './usage-opencode.mjs';

/** Bump to invalidate every cached entry wholesale.
* v2: cached records carry `active` sub-intervals for the idle-gap split.
Expand All @@ -52,8 +53,12 @@ import { readCodexState } from './codex-state.mjs';
* and `rateLimits` (the LAST rate-limit snapshot embedded in the
* rollout's token_count events). A v5-cached Codex session carries
* neither and must be re-derived, or the Limits history reads as empty
* for exactly the sessions that have data. */
export const SCHEMA_VERSION = 6;
* for exactly the sessions that have data.
* v7: the opencode transcript source (usage-opencode.mjs) joins the index —
* SQLite-backed session/message/part rows mapped to the same record
* shape, with opencode's OWN metered cost carried as observed truth
* (`costObserved` on usage rows, preferred over the pricing table). */
export const SCHEMA_VERSION = 7;

/** Silence longer than this ends a stretch of engagement. A session is split
* into active sub-intervals at gaps ABOVE this bound (exactly this much is not
Expand Down Expand Up @@ -271,14 +276,15 @@ export function projectLabel(cwd, dirName) {
return { project: parts.length ? parts[parts.length - 1] : 'unknown', worktree: null };
}

/** Sum a record's per-model usage rows into one API-equivalent cost. */
/** Sum a record's per-model usage rows into one API-equivalent cost. Rows with
* an observed transcript cost (opencode) use it — same preference as aggregate. */
function sessionCost(rec, deps) {
let cost = 0;
for (const row of rec.usage ?? []) {
cost += deps.costOf({
cost += row.costObserved != null ? row.costObserved : (deps.costOf({
model: row.model, provider: rec.provider,
input: row.input, output: row.output, cacheRead: row.cacheRead, cacheWrite: row.cacheWrite,
}) || 0;
}) || 0);
}
return round(cost);
}
Expand Down Expand Up @@ -727,6 +733,15 @@ function codexIdFromName(name) {
}

function parseFile(entry) {
if (entry.provider === 'opencode') {
try {
const parsed = parseOpencodeSession({ dbFile: entry.dbFile, id: entry.id });
// Title hygiene matches the JSONL parsers: the cached index lands on
// disk, so the same secrets mask applies here.
if (parsed?.session) parsed.session.title = maskSecrets(parsed.session.title);
return parsed;
} catch { return null; } // a parser bug must not cost the user their whole index
}
let raw;
try { raw = fs.readFileSync(entry.file, 'utf8'); } catch { return null; }
try {
Expand Down Expand Up @@ -886,10 +901,14 @@ function aggregate(records, { days, now, cutoff, deps }) {
// period ending 2026-09-01) must not retroactively restate a finished
// window — August's spend was metered at August's rate and has to keep
// reading that way. Rows are already keyed by day, so this costs nothing.
const rowCost = deps.costOf({
// costObserved (opencode): the transcript's OWN metered figure for the
// row, when present, outranks the pricing table — observed truth beats a
// rate ak must guess (kimi/openrouter/local). null means no observation
// and the table applies, never a fabricated $0.
const rowCost = row.costObserved != null ? row.costObserved : (deps.costOf({
model: row.model, provider: rec.provider, day: row.day,
input: row.input, output: row.output, cacheRead: row.cacheRead, cacheWrite: row.cacheWrite,
}) || 0;
}) || 0);
input += row.input; output += row.output;
cacheRead += row.cacheRead; cacheWrite += row.cacheWrite;
cost += rowCost;
Expand Down Expand Up @@ -1069,7 +1088,7 @@ function notify(onProgress, payload) {
* @property {number} [days] window size in days (default 14)
* @property {boolean} [force] ignore cached per-file entries
* @property {Function} [onProgress] called with { scanned, total, phase }
* @property {{claude?: string, codex?: string}} [roots] override transcript roots (tests)
* @property {{claude?: string, codex?: string, opencode?: string}} [roots] override transcript roots (tests; opencode = the SQLite store path)
* @property {string} [cachePath] override the index cache location (tests)
* @property {number} [now] override "now" (tests)
* @property {number} [maxAgeMs] readIndex only: memo TTL
Expand Down Expand Up @@ -1108,6 +1127,20 @@ async function scan(o = {}) {
.map((e) => ({ ...e, stat: statSafe(e.file) }))
.filter((e) => e.stat && e.stat.mtimeMs >= cutoff);

// opencode transcript source (one SQLite store, per-session cache keys).
// Same hermeticity rule as the codex ledger below: overridden roots imply
// the REAL store is the wrong one — only default-root scans (or an explicit
// roots.opencode path) read it.
const ocDb = o.roots === undefined ? defaultOpencodeDbPath() : (roots?.opencode ?? null);
if (ocDb && fs.existsSync(ocDb)) {
for (const e of listOpencodeSessions({ dbFile: ocDb, cutoffMs: cutoff })) {
candidates.push({
file: `opencode://${e.id}`, provider: 'opencode', id: e.id, dbFile: ocDb,
stat: { mtimeMs: e.mtimeMs, size: e.size },
});
}
}

const cache = force ? null : readCache(cacheFile);
const entries = {};
const records = [];
Expand All @@ -1124,7 +1157,7 @@ async function scan(o = {}) {
session = parsed ? parsed.session : null;
}
if (session) {
entries[c.file] = { ...key, session };
entries[c.file] = { ...key, session, ...(c.dbFile ? { dbFile: c.dbFile } : {}) };
records.push(session);
}
scanned++;
Expand All @@ -1143,7 +1176,13 @@ async function scan(o = {}) {
const lastActivity = e.session.end ?? e.session.start;
// No timestamp at all → can't judge age; keep it rather than guess.
if (lastActivity != null && now - lastActivity > KEEP_MS) continue;
if (statSafe(file)) entries[file] = e;
// opencode pseudo-keys are not files: existence means "row still in the store".
if (file.startsWith('opencode://')) {
const dbFile = e.dbFile ?? ocDb;
if (dbFile && opencodeSessionExists({ dbFile, id: file.slice('opencode://'.length) })) {
entries[file] = { ...e, dbFile };
}
} else if (statSafe(file)) entries[file] = e;
}
}
writeCache(cacheFile, { schemaVersion: SCHEMA_VERSION, updatedAt: new Date(now).toISOString(), entries });
Expand Down Expand Up @@ -1271,6 +1310,19 @@ function locate(id, r, cacheFile) {
export async function readSession(id, o = {}) {
if (typeof id !== 'string' || !VALID_ID.test(id)) throw invalidId(id);
const r = { ...defaultRoots(), ...(o.roots ?? {}) };

// opencode sessions live in the SQLite store, not a JSONL file — resolve
// them before the file-locating path (pseudo-key opencode://<id>).
const ocDb = o.roots === undefined ? defaultOpencodeDbPath() : (o.roots?.opencode ?? null);
if (ocDb && fs.existsSync(ocDb) && opencodeSessionExists({ dbFile: ocDb, id })) {
const parsed = parseOpencodeSession({ dbFile: ocDb, id, withTurns: true });
if (parsed) {
const rec = parsed.session;
rec.title = maskSecrets(rec.title);
return sessionPayload(rec, parsed.turns);
}
}

const found = locate(id, r, o.cachePath ?? defaultCachePath());
if (!found) return null;

Expand Down Expand Up @@ -1308,8 +1360,13 @@ export async function readSession(id, o = {}) {
: parseClaude(raw, { id, dirName: found.dirName, withTurns: true });
} catch { return null; }

const rec = parsed.session;
const usage = rec.usage.reduce((a, row) => ({
return sessionPayload(parsed.session, parsed.turns, await loadDeps(o.deps));
}

/** The /api/session payload for any parsed record (claude, codex, opencode):
* meta with pricer-backed cost, and secret-masked, truncation-signalled turns. */
function sessionPayload(rec, turns, deps) {
const usage = (rec.usage ?? []).reduce((a, row) => ({
input: a.input + row.input, output: a.output + row.output,
cacheRead: a.cacheRead + row.cacheRead, cacheWrite: a.cacheWrite + row.cacheWrite,
}), { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 });
Expand All @@ -1334,7 +1391,7 @@ export async function readSession(id, o = {}) {
// left undefined: the transcript header rendered a hardcoded "$0.00" on a
// panel whose whole subject is cost. `.filter(Boolean)` could not drop it
// because fmtUsd(undefined) is the truthy string "$0.00".
cost: sessionCost(rec, await loadDeps(o.deps)),
cost: sessionCost(rec, deps),
...usage, tokens: usage.input + usage.output + usage.cacheRead + usage.cacheWrite,
},
// ADR-0009 §8: truncation is the other way content is withheld, and it used
Expand All @@ -1344,7 +1401,7 @@ export async function readSession(id, o = {}) {
// turn cannot be misread as an abridged one. `originalChars` is measured
// after `maskSecrets`, so it describes loss due to truncation alone — it is
// not a raw-file length, and must not be rendered as one.
turns: parsed.turns.map((t) => {
turns: (turns ?? []).map((t) => {
const text = maskSecrets(t.text);
const originalChars = text.length;
if (originalChars <= MAX_TURN_CHARS) return { ...t, text };
Expand Down
Loading
Loading