diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index 46ea08361..17ffe2dce 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -91,6 +91,16 @@ In TUI chat mode there is no completion gate — the session stays open across t - The default user-global settings path, the per-repo `.corbits/settings.json`, and the project/global grant store (`.corbits/permissions.json`) are on the static secret-guard denylist for path-keyed tools, so the agent cannot `read_file` credentials there or persist standing auto-approvals. An arbitrary path selected with `--config ` is not added to that denylist at runtime. In normal and auto modes, shell commands that reference a statically protected path require explicit operator approval; yolo/skip-permissions allows those shell references after catastrophic authorization checks, while path-keyed access to statically protected paths remains hard-denied. - Credential-surface ownership: each auth store module enumerates its own files (`*_AUTH_FILENAME` / `MCP_AUTH_DIRNAME`), the data-only registry in `src/auth/credential-surface.ts` turns them into denylist patterns, `secret-guard-plugin.ts` owns matching (lexical plus realpath), and `@mention` resolution consumes the resolved check — never the registry directly. A new `*-auth.json` token store is denied only once its store module exports its filename and the registry lists it; the coverage test scans store sources for `*-auth.json` literals (registered dirnames get an includes-check instead) and fails the build until both exist. +### Inference credential recovery + +Every inference source registers both live credential material and explicit provenance: OAuth credentials identify their provider and named profile, while API-key and keyless sources are classified separately. The harness freezes the complete source at call start, so classification, the one permitted recovery attempt, and any terminal diagnostic retain the exact provider/profile identity that actually failed even if the session changes provider concurrently. + +On the first credential failure for an uncommitted call, the retry policy may refresh an OAuth credential from that same named profile and retry once immediately at the harness boundary. It does not substitute another profile or provider. Credential failures that survive that same-source attempt are terminal to the reactor's silent source-failover path. API-key, keyless, missing-provenance, and later credential failures do not enter OAuth refresh recovery. + +OAuth stores serialize refresh writes and treat the value observed under the store lock as authoritative. A process that loses a compare-and-swap race adopts the concurrent winner, including its complete token set, instead of publishing or continuing with stale loser material. Credential material and refresh tokens are sanitized recursively before reactor events, run records, worker reports, terminal diagnostics, logs, or other diagnostic sinks can receive them. + +The interactive TUI may offer an explicit alternate-provider/model selector only after the same-profile retry also ends in a terminal credential failure. That choice is generation-scoped and consumed once. It may continue the original operator message only when the failed attempt emitted no committing inference event; after any commitment it switches the live provider without replaying the message. `/connect` replaces or adds credentials and `/model` switches the live source explicitly. Exec and fleet workers use the same one-shot same-profile recovery but never open an auth or alternate-provider prompt; an exhausted failure terminates that attempt with a sanitized recovery diagnostic. + ### TUI Runner (`src/tui/runner/`) - Builds a chat-mode agent using the `ChatDirector` diff --git a/docs/IMPLEMENTATION.md b/docs/IMPLEMENTATION.md index c89685079..8c45243d0 100644 --- a/docs/IMPLEMENTATION.md +++ b/docs/IMPLEMENTATION.md @@ -382,6 +382,22 @@ Providers and credentials are read exclusively from settings files: the global ` **Models-first connect.** There is no standalone `/login` command. `/model` opens on a flat **models-only** list (Recent, Favorites, then connected provider/model rows) built by `buildModelsFirstList` (`src/tui/model-picker.ts`); type-to-filter owns printable keys. Selecting a row runs `applyLiveModelSwitch` (`src/session/live-model-switch.ts`) so inference sources, permission-gate identity, grant persistence identity, and advertised tool schemas cut over together. **Alt+A** or `/connect` opens Connect via `addProviderSelectorChoices` (`src/tui/provider-setup.ts`), which lists every first-class kind including Custom — never bare `c` / Ctrl+A, and never in-list “connect →” rows. First-class API-key rows use a named-instance + auth-only form (instance name, key; catalog base URL is display-only); Custom keeps the full manual form. **Alt+F** toggles favorites; recent/favorite pairs live in global settings (`recentModels` / `favoriteModels`). **Alt+D** sets the default via `setDefaultModel` (global `defaultProvider` + that provider's `defaultModel`) plus `persistConnectedSelection` without switching the live session. First-class providers ship from `packages/first-class-providers` (corbits-agnostic defs) and `packages/opencode-go` (Go catalog, auth validate, multi-protocol endpoints, usage). OAuth providers open the existing browser login modal with a named account step; API-key providers share the same multi-instance naming and pre-seed models on save so selection works without restart. Both OAuth and API-key (including Custom) connects share `persistConnectedSelection` in `provider-setup-submit.ts` so project-local provider/model selection is written alongside global credentials. OpenCode Go forces `OPENCODE_GO_BASE_URL` when `opencodeGo` is set so subscription traffic is not billed as Zen PAYG. +### Inference authentication recovery + +`src/config/source-credentials.ts` stores each source's live `CredentialMaterial` beside `SourceCredentialProvenance`: `{ kind: "oauth", provider, profile }`, `{ kind: "api-key" }`, or `{ kind: "keyless" }`. Codex and xAI source builders register the exact named OAuth profile plus any credential-derived identity header in the same material record. The vendored harness resolves that record at send time and shallow-freezes the complete call-start source; retries therefore keep the original source identity rather than consulting mutable session selection. + +`createCorbitsRetryPolicy` normalizes the attempt error before counting credential failures. On ordinal 1 only, an OAuth source calls `refreshSourceCredentialByProvenance` for that exact provider/profile and retries immediately. A refresh error is replaced with a profile-specific `/connect` diagnostic. Non-OAuth credentials, unknown provenance, later credential failures, and a failed post-refresh retry abort. The vendored harness treats the resulting `credential_failure` as terminal instead of feeding it to automatic source failover. + +Codex and xAI refresh sessions use the auth store's locked compare-and-swap update. `updateTokens` returns the authoritative profile observed under the lock; callers replace the complete mutable token object with that winner. If a refresh request loses to a concurrent process, the valid stored winner is adopted rather than overwritten, including removal of optional fields absent from the winner. The in-memory source credential cell also rotates only if the record used to start refresh is still current. + +`sanitizeDiagnosticText` strips terminal controls, replaces exact configured credentials, and scrubs secret-shaped text; `sanitizeDiagnosticValue` applies the same rules recursively. Refresh failures sanitize the refresh token through the error's message, stack, detail, and cause chain before it leaves the auth boundary. TUI, exec, and worker event paths sanitize before run sinks, persistent error records, terminal/UI notices, logs, and parent-facing worker output. + +TUI recovery state lives in `src/tui/runner/credential-recovery.ts`. It arms only for an operator-originated send that observed both the automatic credential retry and a terminal credential failure, and only when another configured provider/model can be assembled. A monotonically increasing generation makes stale accept/cancel actions inert, and acceptance consumes the generation once. The selector excludes the failed provider. It switches the live provider/model and sends a content-less, generation-correlated director continuation only when no committing inference event occurred; after text, tool, or other committed output it switches without replay. Provider/model option IDs are opaque internal identities used to preserve arbitrary provider and model strings; only their labels are UI contract. + +Exec and fleet workers install the same retry policy and sanitization but no recovery selector or credential prompt, including TTY exec. Their exhausted credential failure is terminal and reports how to repair the profile for a later run. In the TUI, `/connect` reauthorizes or adds a profile and refreshes the live catalog, while `/model` switches to a connected provider/model; either is available after a terminal failure. + +This release changes no settings, OAuth-store, or session persistence format and runs no migration. New sessions and resumed sessions resolve provider selection and credentials from the stores as they exist when the process starts or rebuilds; persisted transcripts do not pin old credential material. A Corbits process already running during upgrade still has the old in-memory harness and credential cells and must be restarted to acquire this recovery behavior. + **OpenCode Go multi-protocol.** Selectable ids come from live `GET /zen/go/v1/models` (process-cached; packaged seed when cold or the fetch fails). Byte and model-count caps fail closed as malformed — never a truncated prefix. Protocol routing stays on the local map; unknown live ids use Chat Completions. `buildGoSource` / `resolveGoEndpoint` pick the adapter and base URL per model (not a single provider-wide OpenAI route). When Go is the active provider, subscription usage is fetched for the status bar and omitted on auth/network failure. ### CLI Verbs and Flags diff --git a/docs/TUI.md b/docs/TUI.md index 18ff0da5f..f4dab17a3 100644 --- a/docs/TUI.md +++ b/docs/TUI.md @@ -572,6 +572,12 @@ pair as the default (global `defaultProvider` + that provider's `defaultModel` active, bare `j`/`k` type into the filter rather than moving the highlight — use arrow keys (or the filtered list's navigation) to move. +After an operator-originated send fails authentication, Corbits first attempts one silent OAuth refresh and retry against the same named profile. If that retry also ends in a terminal credential failure, the shell waits until it is idle and may open a dedicated model/provider picker containing only assemblable models from providers other than the failed provider. It does not appear for API-key failures, the first failure before recovery is attempted, non-credential terminal errors, sends not originated by the operator, or when no alternate is available. Escape consumes the recovery offer without changing provider or replaying input. + +Each recovery offer belongs to the failed send's generation. Enter accepts a displayed provider/model identity once; stale, malformed, duplicate, interrupted, cleared, or superseded acceptance is inert. The selected provider/model becomes live. If the failed attempt emitted no assistant text, tool activity, or other committing inference event, Corbits continues the preserved original operator message once on the selected provider. If anything committed, it **never replays the message**: selection only switches the live provider for the next operator action. The internal option IDs that preserve arbitrary provider/model names are opaque implementation details and are never shown as user-facing syntax. + +The terminal failure remains visible and names `/connect` as the path to reauthorize a profile. `/connect` refreshes the provider catalog after successful authorization; `/model` remains available to switch explicitly to any connected provider/model. Neither command retroactively replays committed work. + `/mcp` uses the same longest-first overlay-hint footer as the model picker: **Alt+A** add (omitted while local MCP settings shadow global), **Alt+D** disable, **Alt+R** remove — never bare letters. A remove confirm drops those diff --git a/docs/VENDORING.md b/docs/VENDORING.md index 89710ff79..f22490320 100644 --- a/docs/VENDORING.md +++ b/docs/VENDORING.md @@ -108,11 +108,22 @@ verbatim (including the two upstream deletions, a ledger entry), and the second re-applies each `PATCHES.md` entry with an as-is / adapt / subsumed triage recorded in the ledgers. No entry was subsumed upstream. Upstream replaced inline provider `apiKey` plumbing -with a `credentialId` + credential-cell model; no ledger entry touches -auth so the vendored trees needed no migration, but first-party callers -were migrated to the new model (each built source registers its secret -in `src/config/source-credentials.ts`, handed to the vendored trees as -their resolver). +with a `credentialId` + credential-cell model. At sync time no existing +ledger entry touched auth, so the pristine trees needed no auth-patch +migration; first-party callers were migrated to the new model (each built +source registers its secret in `src/config/source-credentials.ts`, handed +to the vendored trees as their resolver). + +CL-9347 subsequently adds the coupled `harness-ts-auth-recovery` entry in +`vendor/intx-inference/PATCHES.md` and `runtime-ts-auth-recovery-context` in +`vendor/intx-types/PATCHES.md`; it does not change the upstream pin or +retrieval metadata. Together they atomically resolve bearer material and +credential-derived identity headers, freeze the exact call-start source, +expose per-call credential-failure history to the retry policy, preserve a +classified refresh diagnostic on abort, and make an exhausted credential +failure terminal to automatic source failover. Re-syncs must carry or replace +both entries together so retry identity cannot drift from the credential +material used by the attempt. `vendor/intx-workflow-host/workflow-definition-loader.ts` is new in this sync: a second partial-tree path alongside `adapters/`, carrying diff --git a/src/agent/director.test.ts b/src/agent/director.test.ts index 641949187..e751407e8 100644 --- a/src/agent/director.test.ts +++ b/src/agent/director.test.ts @@ -135,6 +135,55 @@ describe("toolSetDigest", () => { }); }); +describe("ChatDirector credential recovery continuation", () => { + const continuation = (generation: number): ReactorInboundEvent => + ({ + type: "message.received", + message: { + ref: { uid: 0, mailbox: "system" }, + headers: { + from: "user@local", + to: ["agent@local"], + date: "2026-09-26T00:00:00.000Z", + messageId: `credential-recovery-${generation}@local`, + interchangeType: "system.credential.refresh", + interchangeCorrelationId: String(generation), + }, + flags: [], + content: "", + signatureStatus: "missing", + }, + }) as ReactorInboundEvent; + + test("consumes one matching armed continuation and rejects stale or repeated delivery", async () => { + const director = createChatDirector("system", [], {}); + const capabilities = makeCapabilities(); + + expect( + actionsArray( + await director.decide(continuation(4), mockState, capabilities), + ), + ).toEqual([{ type: "reply", content: "" }]); + + director.armCredentialRecoveryContinuation(5); + expect( + actionsArray( + await director.decide(continuation(4), mockState, capabilities), + ), + ).toEqual([{ type: "reply", content: "" }]); + expect( + actionsArray( + await director.decide(continuation(5), mockState, capabilities), + ).map((action) => action.type), + ).toEqual(["infer"]); + expect( + actionsArray( + await director.decide(continuation(5), mockState, capabilities), + ), + ).toEqual([{ type: "reply", content: "" }]); + }); +}); + describe("ChatDirector tool-only loop protection", () => { const providerlessPolicy = { providerName: "test-provider" }; diff --git a/src/agent/director.ts b/src/agent/director.ts index a6086f480..f3e453d05 100644 --- a/src/agent/director.ts +++ b/src/agent/director.ts @@ -457,6 +457,8 @@ function applyManageTasksToolCall( // (inference., tool., reactor., fork.). export const CHAT_TASKS_CHANGED_EVENT = "custom.chat.tasks.changed"; export const CHAT_TOOLS_ACTIVATE_EVENT = "custom.chat.tools.activate"; +export const CREDENTIAL_RECOVERY_INTERCHANGE_TYPE = + "system.credential.refresh" as const; export const ChatTasksChangedDataSchema = type({ tasks: TaskSchema.array(), }); @@ -591,6 +593,7 @@ class ChatDirectorImpl extends DefaultDirector { // preserves the queue for the next successful turn instead of desyncing // the host from already-mutated director state. private coordinatorRethrowNoted = false; + private credentialRecoveryGeneration: number | undefined; constructor( systemPrompt: string, @@ -775,6 +778,16 @@ class ChatDirectorImpl extends DefaultDirector { this.clearDenials = clear; } + armCredentialRecoveryContinuation(generation: number): void { + this.credentialRecoveryGeneration = generation; + } + + cancelCredentialRecoveryContinuation(generation: number): void { + if (this.credentialRecoveryGeneration === generation) { + this.credentialRecoveryGeneration = undefined; + } + } + updateToolDefinitions(toolDefinitions: ToolDefinition[]): void { const before = toolSetDigest(this._toolDefinitions); const after = toolSetDigest(toolDefinitions); @@ -991,6 +1004,23 @@ class ChatDirectorImpl extends DefaultDirector { const recovery = this.compaction.interceptOverflow(event, capabilities); if (recovery !== null) return recovery; + if ( + event.type === "message.received" && + event.message.ref?.mailbox === "system" && + event.message.headers?.interchangeType === + CREDENTIAL_RECOVERY_INTERCHANGE_TYPE + ) { + const generation = Number(event.message.headers.interchangeCorrelationId); + if ( + Number.isSafeInteger(generation) && + generation === this.credentialRecoveryGeneration + ) { + this.credentialRecoveryGeneration = undefined; + return capabilities.infer(); + } + return capabilities.wait(); + } + // A forged or replayed compaction continuation arrives as an empty // message.received with no outstanding compact state (the legit resume // is consumed above). Answering it with infer would burn a billable @@ -1437,6 +1467,8 @@ export interface ChatDirector extends ReactorDirector { setWorkflowCoordinator(coordinator: WorkflowCoordinator | undefined): void; setAllowIdleWithFleet(value: boolean): void; setClearDenials(clear: (() => void) | undefined): void; + armCredentialRecoveryContinuation(generation: number): void; + cancelCredentialRecoveryContinuation(generation: number): void; getTasks(): Task[]; restoreTasks(tasks: Task[]): void; getContextEstimate(): { tokens: number; isEstimate: boolean }; diff --git a/src/agent/prompts.ts b/src/agent/prompts.ts index 2b5637708..3049437ac 100644 --- a/src/agent/prompts.ts +++ b/src/agent/prompts.ts @@ -452,6 +452,15 @@ export function buildSkillsSection(skills: readonly SkillSummary[]): string { ].join("\n"); } +export function buildCorbitsRecoveryCommands(): string { + return [ + "Corbits recovery commands:", + "- For provider authentication, login, reauthentication, or credential failures, recommend /connect and name the provider/profile.", + "- To switch the active provider or model, recommend /model.", + "- For OAuth profiles, never recommend Codex CLI login or API-key setup, and never request or expose secrets.", + ].join("\n"); +} + export function buildChatSystemPrompt( extensions?: string[], env?: EnvironmentInfo, @@ -479,6 +488,7 @@ export function buildChatSystemPrompt( coreToolNamesForSessionMode(sessionMode, toolAvailability), { advertiseArchive: true }, ), + buildCorbitsRecoveryCommands(), ]; if (skills.length > 0) sections.push(buildSkillsSection(skills)); sections.push(contextSection(env)); diff --git a/src/agent/retry-policy.test.ts b/src/agent/retry-policy.test.ts index fe0dfac65..aee14f46a 100644 --- a/src/agent/retry-policy.test.ts +++ b/src/agent/retry-policy.test.ts @@ -1,9 +1,20 @@ import { describe, expect, test } from "bun:test"; +import { createDefaultScheduler, runInference } from "@intx/inference"; +import { createInferenceDependencies } from "../provider/inference-dependencies.js"; +import type { + ConversationTurn, + InferenceEvent, + InferenceSource, +} from "@intx/types/runtime"; import type { AdmissionQueue } from "../subagent/admission.js"; import { createCorbitsRetryPolicy, type CorbitsRetryPolicyOptions, } from "./retry-policy.js"; +import { + clearSourceCredentials, + registerSourceCredentialRecord, +} from "../config/source-credentials.js"; const HTML_503 = `503 Service Unavailable Cloudflare`; @@ -21,6 +32,180 @@ function policy(opts: CorbitsRetryPolicyOptions = {}) { } describe("createCorbitsRetryPolicy", () => { + test("refreshes the first credential failure after a transient retry", async () => { + registerSourceCredentialRecord("codex/work", { + provenance: { kind: "oauth", provider: "codex", profile: "work" }, + material: { + secret: "old", + headers: { "chatgpt-account-id": "old-account" }, + }, + }); + try { + let refreshes = 0; + const decide = policy({ + refreshCredential: async () => { + refreshes++; + }, + }); + const source = { + id: "codex/work", + provider: "codex-responses", + baseURL: "https://chatgpt.com/backend-api/codex", + credentialId: "codex/work", + model: "gpt-5", + }; + + expect( + await decide({ + attempt: 1, + elapsedMs: 0, + source, + credentialFailureOrdinal: 0, + credentialFailureHistory: [], + error: { category: "retryable", message: "gateway unavailable" }, + }), + ).toEqual({ kind: "retry", delayMs: 500 }); + expect( + await decide({ + attempt: 2, + elapsedMs: 500, + source, + credentialFailureOrdinal: 1, + credentialFailureHistory: [], + error: { category: "credential_failure", message: "expired" }, + }), + ).toEqual({ kind: "retry", delayMs: 0 }); + expect(refreshes).toBe(1); + } finally { + clearSourceCredentials(); + } + }); + + test("raw Codex 404 invalid_token refreshes once and surfaces normalized credential failure", async () => { + const source: InferenceSource = { + id: "codex/work", + provider: "codex-responses", + baseURL: "https://chatgpt.com/backend-api/codex", + credentialId: "codex/work", + model: "gpt-5", + }; + registerSourceCredentialRecord(source.credentialId, { + provenance: { kind: "oauth", provider: "codex", profile: "work" }, + material: { secret: "access-token" }, + }); + const oldSecret = "opaque old credential with spaces"; + const replacementSecret = "opaque replacement credential with spaces"; + let liveSecret = oldSecret; + let sends = 0; + let refreshes = 0; + const retryPolicy = policy({ + providerId: () => "xai/previous", + refreshCredential: async (refreshedSource, provenance) => { + refreshes++; + liveSecret = replacementSecret; + expect(refreshedSource.id).toBe(source.id); + expect(provenance).toEqual({ + kind: "oauth", + provider: "codex", + profile: "work", + }); + }, + }); + const baseDeps = await createInferenceDependencies(); + const deps = { + ...baseDeps, + scheduler: createDefaultScheduler(), + fetch: async (_input: string | URL | Request, init?: RequestInit) => { + sends++; + const authorization = new Headers(init?.headers).get("authorization"); + return Response.json( + { + error: { + code: "invalid_token", + message: `The access token ${authorization} has been revoked`, + type: "invalid_request_error", + }, + }, + { status: 404 }, + ); + }, + }; + const turns: ConversationTurn[] = [ + { + role: "user", + content: [{ type: "text", text: "hello" }], + timestamp: 0, + }, + ]; + const events: InferenceEvent[] = []; + let seq = 0; + try { + for await (const event of runInference({ + turns, + source, + nextSeq: () => ++seq, + deps, + readMaterial: () => ({ secret: liveSecret }), + inferenceOptions: { retryPolicy }, + })) + events.push(event); + + expect(JSON.stringify(events)).not.toContain(oldSecret); + expect(JSON.stringify(events)).not.toContain(replacementSecret); + expect(sends).toBe(2); + expect(refreshes).toBe(1); + const retries = events.filter( + (event) => event.type === "inference.retry", + ); + expect(retries).toHaveLength(1); + if (retries[0]?.type !== "inference.retry") + throw new Error("expected inference.retry"); + expect(retries[0].data.previousError.category).toBe("credential_failure"); + const terminal = events.findLast( + (event) => event.type === "inference.error", + ); + if (terminal?.type !== "inference.error") + throw new Error("expected terminal inference.error"); + expect(terminal.data.error.category).toBe("credential_failure"); + expect(terminal.data.error.message).toContain('Codex profile "work"'); + expect(terminal.data.error.message).toContain("/connect"); + } finally { + clearSourceCredentials(); + } + }); + + test("does not recover namespaced API-key credentials", async () => { + registerSourceCredentialRecord("xai/shadow", { + provenance: { kind: "api-key" }, + material: { secret: "explicit-key" }, + }); + try { + let refreshes = 0; + const decision = await policy({ + refreshCredential: async () => { + refreshes++; + }, + })({ + attempt: 1, + elapsedMs: 0, + source: { + id: "xai/shadow", + provider: "openai-compatible", + baseURL: "https://relay.example/v1", + credentialId: "xai/shadow", + model: "relay-model", + }, + credentialFailureOrdinal: 1, + credentialFailureHistory: [], + error: { category: "credential_failure", message: "bad key" }, + }); + expect(decision).toEqual({ kind: "abort" }); + expect(refreshes).toBe(0); + } finally { + clearSourceCredentials(); + } + }); + test("retries protocol_mismatch when the body is an HTML 503 gateway page", async () => { const decision = await policy()({ attempt: 1, diff --git a/src/agent/retry-policy.ts b/src/agent/retry-policy.ts index 1c517f349..e5a93f17d 100644 --- a/src/agent/retry-policy.ts +++ b/src/agent/retry-policy.ts @@ -1,5 +1,6 @@ import { createDefaultRetryPolicy } from "@intx/inference"; import type { + InferenceError, RetryDecision, RetryPolicy, RetrySituation, @@ -12,6 +13,12 @@ import { getProcessAdmissionQueue, type AdmissionQueue, } from "../subagent/admission.js"; +import { refreshSourceCredentialByProvenance } from "../auth/refresh-source-credential.js"; +import { + readSourceCredentialRecord, + type SourceCredentialProvenance, +} from "../config/source-credentials.js"; +import type { InferenceSource } from "@intx/types/runtime"; // Providers that enforce long-window quotas (e.g. monthly limits) set // Retry-After to days or weeks. The default policy trusts that value and @@ -24,14 +31,17 @@ const RATE_LIMIT_HANG_MS = 86_400_000; export interface CorbitsRetryPolicyOptions { /** - * Catalog provider id (e.g. xai/thegreataxios) stamped onto errors before - * normalize. Pass a getter when the live provider can change mid-session - * (e.g. `/model`); it is resolved on each retry decision. + * Fallback catalog provider id for callers that do not supply a source in + * RetrySituation. Harness calls use the frozen call-start source instead. */ providerId?: string | (() => string | undefined); /** Process admission controller. Tests inject a stub; production omits. */ admission?: AdmissionQueue; now?: () => number; + refreshCredential?: ( + source: Readonly, + provenance: Extract, + ) => Promise; } /** @@ -46,24 +56,74 @@ export function createCorbitsRetryPolicy( const defaultPolicy = createDefaultRetryPolicy(); const admission = options?.admission ?? getProcessAdmissionQueue(); const now = options?.now ?? Date.now; - return ( + const normalizeError = ( + incoming: InferenceError, + source?: Readonly, + ): InferenceError => { + const configuredProvider = options?.providerId; + const stampedProviderId = + source?.id ?? + (typeof configuredProvider === "function" + ? configuredProvider() + : configuredProvider); + const contextual = incoming as InferenceErrorWithGoContext; + const withProvider: InferenceErrorWithGoContext = + stampedProviderId !== undefined && contextual.providerId === undefined + ? { ...contextual, providerId: stampedProviderId } + : contextual; + return normalizeInferenceErrorForRetry(withProvider); + }; + const policy = ( situation: RetrySituation, ): RetryDecision | Promise => { - const raw = options?.providerId; - const stampedProviderId = typeof raw === "function" ? raw() : raw; - const incoming = situation.error as InferenceErrorWithGoContext; - const withProvider: InferenceErrorWithGoContext = - stampedProviderId !== undefined && incoming.providerId === undefined - ? { ...incoming, providerId: stampedProviderId } - : incoming; - const error = normalizeInferenceErrorForRetry(withProvider); + const error = normalizeError(situation.error, situation.source); + if ( + error.category === "credential_failure" && + situation.credentialFailureOrdinal === 1 && + situation.source !== undefined + ) { + let provenance: SourceCredentialProvenance; + try { + provenance = readSourceCredentialRecord( + situation.source.credentialId, + ).provenance; + } catch { + return { kind: "abort" }; + } + if (provenance.kind === "oauth") { + const refresh = options?.refreshCredential; + const pending = + refresh !== undefined + ? refresh(situation.source, provenance) + : refreshSourceCredentialByProvenance( + situation.source.credentialId, + ).then(() => undefined); + return pending.then( + () => ({ kind: "retry", delayMs: 0 }), + (cause: unknown) => ({ + kind: "abort", + error: { + category: "credential_failure", + providerId: situation.source?.id, + message: `${provenance.provider} profile "${provenance.profile}" could not be refreshed${cause instanceof Error ? `: ${cause.message}` : ""}. Run /connect, choose ${provenance.provider}, and reconnect profile "${provenance.profile}".`, + }, + }), + ); + } + } if (error.category === "retryable" && error.statusCode === 429) { const pauseMs = Math.min( error.retryAfterMs ?? DEFAULT_PRESSURE_PAUSE_MS, MAX_BLIND_WAIT_MS, ); + const configuredProvider = options?.providerId; const provider = - withProvider.providerId ?? stampedProviderId ?? "unknown"; + (situation.error as InferenceErrorWithGoContext).providerId ?? + situation.source?.id ?? + (typeof configuredProvider === "function" + ? configuredProvider() + : configuredProvider) ?? + "unknown"; admission.notePressure(provider, now() + pauseMs); // The vendored default retries `retryable` on a fixed 500/1000ms // schedule and ignores Retry-After. A 429 carries the server's pacing @@ -93,4 +153,5 @@ export function createCorbitsRetryPolicy( } return defaultPolicy({ ...situation, error }); }; + return Object.assign(policy, { normalizeError }); } diff --git a/src/auth/codex/session-failure.test.ts b/src/auth/codex/session-failure.test.ts index 02aa326a8..1a3ed1778 100644 --- a/src/auth/codex/session-failure.test.ts +++ b/src/auth/codex/session-failure.test.ts @@ -2,12 +2,20 @@ import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, test } from "bun:test"; +import { + OAuthRefreshFailedError, + OAuthTokenEndpointError, +} from "@corbits/oauth-core"; +import { errorMessage } from "../../agent/error-message.js"; import { saveCodexProfile } from "../../config/oauth-stores.js"; +import { formatSubAgentSpawnAuthFailureMessage } from "../../subagent/inference-auth-failure.js"; import { codexAuthFailureDiagnostic, CodexAuthError, CodexRefreshLockError, + createCodexTokenSession, getValidCodexToken, + refreshStagedCodexTokens, } from "./session.js"; async function tempHome(): Promise { @@ -55,7 +63,7 @@ describe("codex auth failure surface", () => { globalThis.fetch = (async () => new Response(JSON.stringify({ error: "invalid_grant" }), { status: 400, - })) as unknown as typeof fetch; + })) as unknown as unknown as typeof fetch; try { let failure: unknown; try { @@ -72,6 +80,101 @@ describe("codex auth failure surface", () => { } }); + test("a token endpoint cannot reflect the stored refresh credential", async () => { + const home = await tempHome(); + const now = Date.now(); + const refresh = "opaque refresh value / with spaces?!"; + await saveCodexProfile( + { + name: "shared", + createdAt: now, + tokens: { + access: "access-1", + refresh, + expiresAt: now - 300_000, + }, + }, + home, + ); + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => + new Response(`grant rejected for ${refresh}`, { + status: 400, + })) as unknown as typeof fetch; + try { + const failure = await getValidCodexToken("shared", now, home).catch( + (error: unknown) => error, + ); + expect(failure).toBeInstanceOf(CodexAuthError); + const auth = failure as CodexAuthError; + const surfaced = JSON.stringify({ + auth: String(auth), + retry: { + type: "inference.retry", + data: { previousError: { message: auth.message } }, + }, + terminal: { + type: "inference.error", + data: { error: { message: auth.message } }, + }, + log: errorMessage(auth), + guidance: formatSubAgentSpawnAuthFailureMessage("auth task", auth), + }); + expect(surfaced).not.toContain(refresh); + expect(surfaced).toContain("grant rejected"); + expect(surfaced).toContain("Re-authenticate"); + } finally { + globalThis.fetch = originalFetch; + await rm(home, { recursive: true, force: true }); + } + }); + + test("normal and staged endpoint failures sanitize every diagnostic projection", async () => { + const home = await tempHome(); + const now = Date.now(); + const refresh = "opaque codex refresh / reflected?!"; + const tokens = { + access: "access-1", + refresh, + expiresAt: now - 300_000, + }; + await saveCodexProfile({ name: "shared", createdAt: now, tokens }, home); + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => + new Response(`grant rejected for ${refresh}`, { + status: 401, + })) as unknown as typeof fetch; + try { + const normal = await createCodexTokenSession(home) + .getValidToken("shared", now) + .catch((error: unknown) => error); + expect(normal).toBeInstanceOf(OAuthRefreshFailedError); + const normalCause = (normal as OAuthRefreshFailedError).cause; + expect(normalCause).toBeInstanceOf(OAuthTokenEndpointError); + expect(normalCause).toMatchObject({ status: 401 }); + + const staged = await refreshStagedCodexTokens({ ...tokens }, now).catch( + (error: unknown) => error, + ); + expect(staged).toBeInstanceOf(OAuthTokenEndpointError); + expect(staged).toMatchObject({ status: 401 }); + + for (const failure of [normal, staged]) { + let current: unknown = failure; + while (current instanceof Error) { + expect(current.message).not.toContain(refresh); + expect(current.stack).not.toContain(refresh); + if (current instanceof OAuthTokenEndpointError) + expect(current.detail).not.toContain(refresh); + current = current.cause; + } + } + } finally { + globalThis.fetch = originalFetch; + await rm(home, { recursive: true, force: true }); + } + }); + test("fresh tokens resolve without touching the network", async () => { const home = await tempHome(); const now = Date.now(); @@ -91,7 +194,7 @@ describe("codex auth failure surface", () => { const originalFetch = globalThis.fetch; globalThis.fetch = (async () => { throw new Error("network must not be touched for fresh tokens"); - }) as unknown as typeof fetch; + }) as unknown as unknown as typeof fetch; try { const access = await getValidCodexToken("shared", now, home); expect(access.access).toBe("access-1"); diff --git a/src/auth/codex/session-refresh-race.test.ts b/src/auth/codex/session-refresh-race.test.ts index 497d06468..6ebdbaab5 100644 --- a/src/auth/codex/session-refresh-race.test.ts +++ b/src/auth/codex/session-refresh-race.test.ts @@ -2,7 +2,10 @@ import { mkdtemp, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { describe, expect, test } from "bun:test"; -import { saveCodexProfile } from "../../config/oauth-stores.js"; +import { + loadCodexProfile, + saveCodexProfile, +} from "../../config/oauth-stores.js"; import { createCodexTokenSession } from "./session.js"; // Two concurrent headless runs share one Codex credential: two independent @@ -12,6 +15,64 @@ import { createCodexTokenSession } from "./session.js"; // invalid_grant. Both callers must resolve with the rotated access token and // the endpoint must see a single grant. describe("codex shared-credential refresh race", () => { + test("a concurrent winner without accountId replaces the loser exactly", async () => { + const home = await mkdtemp(join(tmpdir(), "cl9347-codex-winner-")); + const now = Date.now(); + await saveCodexProfile( + { + name: "shared", + createdAt: now, + tokens: { + access: "access-old", + refresh: "refresh-old", + expiresAt: now - 300_000, + accountId: "old-account", + }, + }, + home, + ); + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => { + await saveCodexProfile( + { + name: "shared", + createdAt: now, + tokens: { + access: "access-winner", + refresh: "refresh-winner", + expiresAt: now + 3_600_000, + }, + }, + home, + ); + return Response.json({ + access_token: "access-loser", + refresh_token: "refresh-loser", + expires_in: 3600, + }); + }) as unknown as typeof fetch; + + try { + const access = await createCodexTokenSession(home).getValidToken( + "shared", + now, + ); + expect(access).toEqual({ access: "access-winner" }); + expect(await loadCodexProfile("shared", home)).toEqual({ + name: "shared", + createdAt: now, + tokens: { + access: "access-winner", + refresh: "refresh-winner", + expiresAt: now + 3_600_000, + }, + }); + } finally { + globalThis.fetch = originalFetch; + await rm(home, { recursive: true, force: true }); + } + }); + test("concurrent refreshes on one shared store both resolve with one grant", async () => { const home = await mkdtemp(join(tmpdir(), "cl8628-race-")); const now = Date.now(); @@ -61,7 +122,7 @@ describe("codex shared-credential refresh race", () => { refresh_token: "refresh-2", expires_in: 3600, }); - }) as typeof fetch; + }) as unknown as typeof fetch; try { const first = createCodexTokenSession(home); diff --git a/src/auth/codex/session.ts b/src/auth/codex/session.ts index 252ea1738..4431bcade 100644 --- a/src/auth/codex/session.ts +++ b/src/auth/codex/session.ts @@ -17,6 +17,10 @@ import { updateCodexTokens, } from "../../config/oauth-stores.js"; import type { InferenceErrorLike } from "../../inference-gateway-error.js"; +import { + replaceMutableTokens, + sanitizedRefreshFailure, +} from "../token-session-boundary.js"; import { CodexRefreshLockTimeoutError, withCodexRefreshLock, @@ -110,26 +114,49 @@ async function refreshCodexTokensForStore( export function createCodexTokenSession( home?: string, ): TokenSession { + const refreshBasis = new WeakMap(); const inner = createTokenSession({ skewMs: CODEX_REFRESH_SKEW_MS, loadProfile: (name) => loadCodexProfile(name, home), - updateTokens: (name, tokens) => updateCodexTokens(name, tokens, home), + updateTokens: async (name, tokens) => { + const winner = await updateCodexTokens( + name, + tokens, + home, + refreshBasis.get(tokens), + ); + if (winner === undefined) throw new OAuthProfileNotFoundError(name); + replaceMutableTokens(tokens, winner.tokens); + }, // createTokenSession only passes (refresh, now). The package refresh // helper needs prior tokens to keep chatgpt-account-id; mergeRefreshed // supplies that after this stub call. - refreshTokens: (refreshToken, now) => - refreshCodexTokensForStore(refreshToken, now, { - access: "", - refresh: refreshToken, - }), + refreshTokens: async (refreshToken, now) => { + try { + const refreshed = await refreshCodexTokensForStore(refreshToken, now, { + access: "", + refresh: refreshToken, + }); + refreshBasis.set(refreshed, refreshToken); + return refreshed; + } catch (error) { + throw sanitizedRefreshFailure(error, refreshToken); + } + }, toAccess: (tokens) => ({ access: tokens.access, accountId: tokens.accountId, }), - mergeRefreshed: (refreshed, previous) => - refreshed.accountId === undefined && previous.accountId !== undefined - ? { ...refreshed, accountId: previous.accountId } - : refreshed, + mergeRefreshed: (refreshed, previous) => { + const merged = + refreshed.accountId === undefined && previous.accountId !== undefined + ? { ...refreshed, accountId: previous.accountId } + : refreshed; + const expectedRefreshToken = refreshBasis.get(refreshed); + if (expectedRefreshToken !== undefined) + refreshBasis.set(merged, expectedRefreshToken); + return merged; + }, }); return { isExpired: inner.isExpired, @@ -219,11 +246,12 @@ export async function refreshStagedCodexTokens( now: number = Date.now(), ): Promise { if (!isCodexTokenExpired(tokens, now)) return tokens; - const refreshed = await refreshCodexTokensForStore( - tokens.refresh, - now, - tokens, - ); - Object.assign(tokens, refreshed); + let refreshed: CodexTokens; + try { + refreshed = await refreshCodexTokensForStore(tokens.refresh, now, tokens); + } catch (error) { + throw sanitizedRefreshFailure(error, tokens.refresh); + } + replaceMutableTokens(tokens, refreshed); return tokens; } diff --git a/src/auth/refresh-source-credential.ts b/src/auth/refresh-source-credential.ts new file mode 100644 index 000000000..0238e1974 --- /dev/null +++ b/src/auth/refresh-source-credential.ts @@ -0,0 +1,49 @@ +import { xaiUserIdFromAccessToken } from "@corbits/xai-provider"; +import type { CredentialMaterial } from "@intx/types"; + +import { + findSourceCredentialRecord, + rotateSourceCredentialMaterialIfCurrent, + type SourceCredentialProvenance, +} from "../config/source-credentials.js"; +import { getValidCodexToken } from "./codex/session.js"; +import { getValidXaiToken } from "./xai/session.js"; + +export type OAuthCredentialProvenance = Extract< + SourceCredentialProvenance, + { kind: "oauth" } +>; + +async function resolveOAuthCredentialMaterial( + provenance: OAuthCredentialProvenance, +): Promise { + if (provenance.provider === "codex") { + const fresh = await getValidCodexToken(provenance.profile); + return { + secret: fresh.access, + ...(fresh.accountId !== undefined + ? { headers: { "chatgpt-account-id": fresh.accountId } } + : {}), + }; + } + + const fresh = await getValidXaiToken(provenance.profile); + const userId = xaiUserIdFromAccessToken(fresh.access); + return { + secret: fresh.access, + ...(userId !== undefined ? { headers: { "x-grok-user-id": userId } } : {}), + }; +} + +export async function refreshSourceCredentialByProvenance( + credentialId: string, +): Promise { + const record = findSourceCredentialRecord(credentialId); + if (record?.provenance.kind !== "oauth") return false; + const material = await resolveOAuthCredentialMaterial(record.provenance); + return rotateSourceCredentialMaterialIfCurrent( + credentialId, + record, + material, + ); +} diff --git a/src/auth/store.test.ts b/src/auth/store.test.ts index 945302756..e74b8fa7e 100644 --- a/src/auth/store.test.ts +++ b/src/auth/store.test.ts @@ -209,7 +209,11 @@ describe("createAuthStore", () => { "Timed out waiting for OAuth credential lock", ); await rm(lockPath, { force: true }); - await expect(second).resolves.toBeUndefined(); + await expect(second).resolves.toEqual({ + name: "work", + tokens: { access: "second", refresh: "r2", expiresAt: 3 }, + createdAt: 1, + }); expect((await store.loadProfile("work", home))?.tokens.access).toBe( "second", @@ -249,6 +253,30 @@ describe("createAuthStore", () => { expect(updated?.tokens.access).toBe("a2"); expect(updated?.createdAt).toBe(10); + await store.saveProfile( + { + name: "work", + tokens: { + access: "replacement", + refresh: "new-refresh", + expiresAt: 3, + }, + createdAt: 20, + }, + home, + ); + await store.updateTokens( + "work", + { access: "stale-refresh", refresh: "rotated-old", expiresAt: 4 }, + home, + "r2", + ); + expect(await store.loadProfile("work", home)).toEqual({ + name: "work", + tokens: { access: "replacement", refresh: "new-refresh", expiresAt: 3 }, + createdAt: 20, + }); + await store.updateTokens( "gone", { access: "x", refresh: "x", expiresAt: 0 }, @@ -294,7 +322,7 @@ describe("createAuthStore", () => { await expect( store.updateTokens("work", profile.tokens, home), - ).resolves.toBeUndefined(); + ).resolves.toEqual(profile); } finally { await rm(home, { recursive: true, force: true }); } diff --git a/src/auth/store.ts b/src/auth/store.ts index 4ff85a751..1f587c99e 100644 --- a/src/auth/store.ts +++ b/src/auth/store.ts @@ -29,7 +29,12 @@ export interface AuthStore { home?: string, ) => Promise | undefined>; saveProfile: (profile: AuthProfile, home?: string) => Promise; - updateTokens: (name: string, tokens: TTokens, home?: string) => Promise; + updateTokens: ( + name: string, + tokens: TTokens, + home?: string, + expectedRefreshToken?: string, + ) => Promise | undefined>; // Remove one profile, or all profiles when `name` is undefined. Returns the // names removed. removeProfile: (name: string | undefined, home?: string) => Promise; @@ -221,19 +226,28 @@ export function createAuthStore( await writeAuthFile(file, home); }); }, - // Persist refreshed tokens for an existing profile, preserving createdAt. A - // no-op if the profile no longer exists (e.g. removed in another session). + // Persist refreshed tokens for an existing profile, preserving createdAt. + // The returned profile is the authoritative value observed under the lock: + // either this update, a concurrent winner, or undefined after removal. async updateTokens( name: string, tokens: TTokens, home: string = homedir(), - ): Promise { - await enqueueAuthFileOp(home, async () => { + expectedRefreshToken?: string, + ): Promise | undefined> { + return enqueueAuthFileOp(home, async () => { const file = await readAuthFile(home); const existing = file.profiles[name]; - if (existing === undefined) return; - file.profiles[name] = { ...existing, tokens }; + if (existing === undefined) return undefined; + if ( + expectedRefreshToken !== undefined && + existing.tokens.refresh !== expectedRefreshToken + ) + return existing; + const updated = { ...existing, tokens }; + file.profiles[name] = updated; await writeAuthFile(file, home); + return updated; }); }, async removeProfile( diff --git a/src/auth/token-session-boundary.test.ts b/src/auth/token-session-boundary.test.ts new file mode 100644 index 000000000..3bcb87dee --- /dev/null +++ b/src/auth/token-session-boundary.test.ts @@ -0,0 +1,30 @@ +import { describe, expect, test } from "bun:test"; +import { OAuthTokenEndpointError } from "@corbits/oauth-core"; +import { sanitizedRefreshFailure } from "./token-session-boundary.js"; + +describe("sanitizedRefreshFailure", () => { + test("scrubs already-materialized stacks recursively without losing classification", () => { + const refresh = "opaque refresh credential / reflected?!"; + const endpoint = new OAuthTokenEndpointError( + 401, + `grant rejected for ${refresh}`, + ); + const failure = new Error(`refresh failed for ${refresh}`, { + cause: endpoint, + }); + expect(failure.stack).toContain(refresh); + expect(endpoint.stack).toContain(refresh); + + const sanitized = sanitizedRefreshFailure(failure, refresh); + + expect(sanitized).toBe(failure); + expect(sanitized.message).not.toContain(refresh); + expect(sanitized.stack).not.toContain(refresh); + expect(sanitized.cause).toBe(endpoint); + expect(endpoint).toBeInstanceOf(OAuthTokenEndpointError); + expect(endpoint.status).toBe(401); + expect(endpoint.message).not.toContain(refresh); + expect(endpoint.detail).not.toContain(refresh); + expect(endpoint.stack).not.toContain(refresh); + }); +}); diff --git a/src/auth/token-session-boundary.ts b/src/auth/token-session-boundary.ts new file mode 100644 index 000000000..ecfbbcc5f --- /dev/null +++ b/src/auth/token-session-boundary.ts @@ -0,0 +1,40 @@ +import { sanitizeDiagnosticText } from "../diagnostic-sanitize.js"; + +export function replaceMutableTokens( + target: TTokens, + source: TTokens, +): void { + const replacement = { ...source }; + const mutable = target as Record; + for (const key of Object.keys(mutable)) Reflect.deleteProperty(mutable, key); + Object.assign(target, replacement); +} + +export function sanitizedRefreshFailure( + error: unknown, + refreshToken: string, +): Error { + if (!(error instanceof Error)) + return new Error(sanitizeDiagnosticText(String(error), [refreshToken])); + + Object.defineProperty(error, "message", { + configurable: true, + value: sanitizeDiagnosticText(error.message, [refreshToken]), + }); + if (typeof error.stack === "string") + Object.defineProperty(error, "stack", { + configurable: true, + value: sanitizeDiagnosticText(error.stack, [refreshToken]), + }); + if ("detail" in error && typeof error.detail === "string") + Object.defineProperty(error, "detail", { + configurable: true, + value: sanitizeDiagnosticText(error.detail, [refreshToken]), + }); + if (error.cause !== undefined) + Object.defineProperty(error, "cause", { + configurable: true, + value: sanitizedRefreshFailure(error.cause, refreshToken), + }); + return error; +} diff --git a/src/auth/xai/session-refresh-race.test.ts b/src/auth/xai/session-refresh-race.test.ts new file mode 100644 index 000000000..a7cb6c9a1 --- /dev/null +++ b/src/auth/xai/session-refresh-race.test.ts @@ -0,0 +1,304 @@ +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, test } from "bun:test"; +import { + OAuthRefreshFailedError, + OAuthTokenEndpointError, +} from "@corbits/oauth-core"; +import { errorMessage } from "../../agent/error-message.js"; +import { loadXaiProfile, saveXaiProfile } from "../../config/oauth-stores.js"; +import { formatSubAgentSpawnAuthFailureMessage } from "../../subagent/inference-auth-failure.js"; +import { + createXaiTokenSession, + getValidXaiToken, + XaiAuthError, +} from "./session.js"; + +async function saveExpiredProfile(home: string, now: number): Promise { + await saveXaiProfile( + { + name: "shared", + createdAt: now, + tokens: { + access: "access-1", + refresh: "refresh-1", + expiresAt: now - 300_000, + }, + }, + home, + ); +} + +describe("xAI shared-credential refresh race", () => { + test("a concurrent winner replaces every optional loser field", async () => { + const home = await mkdtemp(join(tmpdir(), "cl9347-xai-winner-")); + const now = Date.now(); + await saveXaiProfile( + { + name: "shared", + createdAt: now, + tokens: { + access: "access-old", + refresh: "refresh-old", + expiresAt: now - 300_000, + idToken: "id-old", + }, + }, + home, + ); + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => { + await saveXaiProfile( + { + name: "shared", + createdAt: now, + tokens: { + access: "access-winner", + refresh: "refresh-winner", + expiresAt: now + 3_600_000, + }, + }, + home, + ); + return Response.json({ + access_token: "access-loser", + refresh_token: "refresh-loser", + expires_in: 3600, + id_token: "id-loser", + }); + }) as unknown as typeof fetch; + + try { + expect( + await createXaiTokenSession(home).getValidToken("shared", now), + ).toEqual({ access: "access-winner" }); + expect(await loadXaiProfile("shared", home)).toEqual({ + name: "shared", + createdAt: now, + tokens: { + access: "access-winner", + refresh: "refresh-winner", + expiresAt: now + 3_600_000, + }, + }); + } finally { + globalThis.fetch = originalFetch; + await rm(home, { recursive: true, force: true }); + } + }); + + test("independent sessions return the committed rotated winner", async () => { + const home = await mkdtemp(join(tmpdir(), "cl9347-xai-race-")); + const now = Date.now(); + await saveExpiredProfile(home, now); + const originalFetch = globalThis.fetch; + let grants = 0; + let resolveSecond!: () => void; + const secondArrived = new Promise((resolve) => { + resolveSecond = resolve; + }); + globalThis.fetch = (async () => { + grants += 1; + if (grants === 1) { + await secondArrived; + return Response.json({ + access_token: "access-2", + refresh_token: "refresh-2", + expires_in: 3600, + }); + } + resolveSecond(); + while ( + (await loadXaiProfile("shared", home))?.tokens.refresh !== "refresh-2" + ) + await new Promise((resolve) => setTimeout(resolve, 5)); + return new Response(JSON.stringify({ error: "invalid_grant" }), { + status: 400, + }); + }) as unknown as typeof fetch; + + try { + const first = createXaiTokenSession(home); + const second = createXaiTokenSession(home); + expect( + await Promise.all([ + first.getValidToken("shared", now), + second.getValidToken("shared", now), + ]), + ).toEqual([{ access: "access-2" }, { access: "access-2" }]); + expect(grants).toBe(2); + } finally { + globalThis.fetch = originalFetch; + await rm(home, { recursive: true, force: true }); + } + }); + + test("independent sessions return a committed non-rotating winner", async () => { + const home = await mkdtemp(join(tmpdir(), "cl9347-xai-static-race-")); + const now = Date.now(); + await saveExpiredProfile(home, now); + const originalFetch = globalThis.fetch; + let grants = 0; + let resolveSecond!: () => void; + const secondArrived = new Promise((resolve) => { + resolveSecond = resolve; + }); + globalThis.fetch = (async () => { + grants += 1; + if (grants === 1) { + await secondArrived; + return Response.json({ + access_token: "access-2", + expires_in: 3600, + }); + } + resolveSecond(); + while ( + (await loadXaiProfile("shared", home))?.tokens.access !== "access-2" + ) + await new Promise((resolve) => setTimeout(resolve, 5)); + return new Response(JSON.stringify({ error: "invalid_grant" }), { + status: 400, + }); + }) as unknown as typeof fetch; + + try { + const first = createXaiTokenSession(home); + const second = createXaiTokenSession(home); + expect( + await Promise.all([ + first.getValidToken("shared", now), + second.getValidToken("shared", now), + ]), + ).toEqual([{ access: "access-2" }, { access: "access-2" }]); + expect(grants).toBe(2); + expect(await loadXaiProfile("shared", home)).toMatchObject({ + tokens: { + access: "access-2", + refresh: "refresh-1", + expiresAt: now + 3_600_000, + }, + }); + } finally { + globalThis.fetch = originalFetch; + await rm(home, { recursive: true, force: true }); + } + }); + + test("invalid_grant without a newer winner remains actionable", async () => { + const home = await mkdtemp(join(tmpdir(), "cl9347-xai-invalid-")); + const now = Date.now(); + await saveExpiredProfile(home, now); + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => + new Response(JSON.stringify({ error: "invalid_grant" }), { + status: 400, + })) as unknown as typeof fetch; + try { + await expect(getValidXaiToken("shared", now, home)).rejects.toMatchObject( + { + name: "XaiAuthError", + reason: "refresh-failed", + message: expect.stringContaining("Log in again"), + } satisfies Partial, + ); + } finally { + globalThis.fetch = originalFetch; + await rm(home, { recursive: true, force: true }); + } + }); + + test("a token endpoint cannot reflect the stored refresh credential", async () => { + const home = await mkdtemp(join(tmpdir(), "cl9347-xai-redact-")); + const now = Date.now(); + const refresh = "opaque refresh value / with spaces?!"; + await saveXaiProfile( + { + name: "shared", + createdAt: now, + tokens: { + access: "access-1", + refresh, + expiresAt: now - 300_000, + }, + }, + home, + ); + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => + new Response(`grant rejected for ${refresh}`, { + status: 400, + })) as unknown as typeof fetch; + try { + const failure = await getValidXaiToken("shared", now, home).catch( + (error: unknown) => error, + ); + expect(failure).toBeInstanceOf(XaiAuthError); + const auth = failure as XaiAuthError; + const surfaced = JSON.stringify({ + auth: String(auth), + retry: { + type: "inference.retry", + data: { previousError: { message: auth.message } }, + }, + terminal: { + type: "inference.error", + data: { error: { message: auth.message } }, + }, + log: errorMessage(auth), + guidance: formatSubAgentSpawnAuthFailureMessage("auth task", auth), + }); + expect(surfaced).not.toContain(refresh); + expect(surfaced).toContain("grant rejected"); + expect(surfaced).toContain("Re-authenticate"); + } finally { + globalThis.fetch = originalFetch; + await rm(home, { recursive: true, force: true }); + } + }); + + test("real endpoint errors retain classification without stack credentials", async () => { + const home = await mkdtemp(join(tmpdir(), "cl9347-xai-stack-")); + const now = Date.now(); + const refresh = "opaque xai refresh / reflected?!"; + await saveXaiProfile( + { + name: "shared", + createdAt: now, + tokens: { + access: "access-1", + refresh, + expiresAt: now - 300_000, + }, + }, + home, + ); + const originalFetch = globalThis.fetch; + globalThis.fetch = (async () => + new Response(`grant rejected for ${refresh}`, { + status: 403, + })) as unknown as typeof fetch; + try { + const failure = await createXaiTokenSession(home) + .getValidToken("shared", now) + .catch((error: unknown) => error); + expect(failure).toBeInstanceOf(OAuthRefreshFailedError); + const cause = (failure as OAuthRefreshFailedError).cause; + expect(cause).toBeInstanceOf(OAuthTokenEndpointError); + expect(cause).toMatchObject({ status: 403 }); + + let current: unknown = failure; + while (current instanceof Error) { + expect(current.message).not.toContain(refresh); + expect(current.stack).not.toContain(refresh); + if (current instanceof OAuthTokenEndpointError) + expect(current.detail).not.toContain(refresh); + current = current.cause; + } + } finally { + globalThis.fetch = originalFetch; + await rm(home, { recursive: true, force: true }); + } + }); +}); diff --git a/src/auth/xai/session.ts b/src/auth/xai/session.ts index 3575c0b80..f18cc2676 100644 --- a/src/auth/xai/session.ts +++ b/src/auth/xai/session.ts @@ -12,6 +12,10 @@ import { } from "@corbits/xai-provider"; import { loadXaiProfile, updateXaiTokens } from "../../config/oauth-stores.js"; +import { + replaceMutableTokens, + sanitizedRefreshFailure, +} from "../token-session-boundary.js"; export class XaiAuthError extends Error { readonly profile: string; @@ -54,17 +58,62 @@ function wrapXaiAuthError(name: string, err: unknown): never { const sessions = new Map>(); -function sessionFor(home?: string): TokenSession { - const key = home ?? ""; - const existing = sessions.get(key); - if (existing !== undefined) return existing; - const created = createTokenSession({ +export function createXaiTokenSession( + home?: string, +): TokenSession { + const refreshBasis = new WeakMap(); + const inner = createTokenSession({ skewMs: XAI_REFRESH_SKEW_MS, loadProfile: (name) => loadXaiProfile(name, home), - updateTokens: (name, tokens) => updateXaiTokens(name, tokens, home), - refreshTokens: refreshXaiTokens, + updateTokens: async (name, tokens) => { + const winner = await updateXaiTokens( + name, + tokens, + home, + refreshBasis.get(tokens), + ); + if (winner === undefined) throw new OAuthProfileNotFoundError(name); + replaceMutableTokens(tokens, winner.tokens); + }, + refreshTokens: async (refreshToken, now) => { + try { + const refreshed = await refreshXaiTokens(refreshToken, now); + refreshBasis.set(refreshed, refreshToken); + return refreshed; + } catch (error) { + throw sanitizedRefreshFailure(error, refreshToken); + } + }, toAccess: (tokens) => ({ access: tokens.access }), }); + return { + isExpired: inner.isExpired, + getValidToken: async (name, now = Date.now()) => { + const basis = await loadXaiProfile(name, home); + try { + return await inner.getValidToken(name, now); + } catch (error) { + if (error instanceof OAuthRefreshFailedError && basis !== undefined) { + const winner = await loadXaiProfile(name, home); + if ( + winner !== undefined && + !inner.isExpired(winner.tokens, now) && + (winner.tokens.access !== basis.tokens.access || + winner.tokens.expiresAt !== basis.tokens.expiresAt) + ) + return { access: winner.tokens.access }; + } + throw error; + } + }, + }; +} + +function sessionFor(home?: string): TokenSession { + const key = home ?? ""; + const existing = sessions.get(key); + if (existing !== undefined) return existing; + const created = createXaiTokenSession(home); sessions.set(key, created); return created; } @@ -90,7 +139,12 @@ export async function refreshStagedXaiTokens( now: number = Date.now(), ): Promise { if (!isXaiTokenExpired(tokens, now)) return tokens; - const refreshed = await refreshXaiTokens(tokens.refresh, now); - Object.assign(tokens, refreshed); + let refreshed: XaiTokens; + try { + refreshed = await refreshXaiTokens(tokens.refresh, now); + } catch (error) { + throw sanitizedRefreshFailure(error, tokens.refresh); + } + replaceMutableTokens(tokens, refreshed); return tokens; } diff --git a/src/config.test.ts b/src/config.test.ts index 24ecd9e32..25389f379 100644 --- a/src/config.test.ts +++ b/src/config.test.ts @@ -1885,6 +1885,7 @@ describe("buildXaiSource", () => { test("omits reasoning_effort when effort is absent", () => { const source = buildXaiSource({ id: "xai/work", + profile: "work", apiKey: "tok", model: "grok-4.6", sessionId: "sess-1", @@ -1898,6 +1899,7 @@ describe("buildXaiSource", () => { test("sets providerOptions.reasoning_effort when effort is present", () => { const source = buildXaiSource({ id: "xai/work", + profile: "work", apiKey: "tok", model: "grok-4.6", sessionId: "sess-1", @@ -1911,6 +1913,7 @@ describe("buildXaiSource", () => { test("does not invent high when effort is absent", () => { const source = buildXaiSource({ id: "xai/work", + profile: "work", apiKey: "tok", model: "grok-4.6", sessionId: "sess-1", @@ -1923,6 +1926,7 @@ describe("buildXaiSource", () => { test("stashes the session id for the adapter's prompt_cache_key", () => { const source = buildXaiSource({ id: "xai/work", + profile: "work", apiKey: "tok", model: "grok-4.6", sessionId: "sess-1", diff --git a/src/config/index.ts b/src/config/index.ts index c99dc9572..ed250d6bc 100644 --- a/src/config/index.ts +++ b/src/config/index.ts @@ -29,7 +29,10 @@ import { import type { CodexProfile } from "../auth/codex/store.js"; import type { XaiProfile } from "../auth/xai/store.js"; import { listCodexProfiles, listXaiProfiles } from "./oauth-stores.js"; -import { registerSourceCredential } from "./source-credentials.js"; +import { + registerSourceCredentialRecord, + type SourceCredentialProvenance, +} from "./source-credentials.js"; import { codexProfilesToCatalogEntries, codexProvidersAsSettings, @@ -47,13 +50,11 @@ import { CODEX_BASE_URL } from "../auth/codex/constants.js"; import { XAI_BASE_URL } from "../auth/xai/constants.js"; import { CODEX_RESPONSES_PROVIDER, - CODEX_ACCOUNT_ID_OPTION, CODEX_SESSION_ID_OPTION, } from "../provider/codex-responses.js"; import { GROK_RESPONSES_PROVIDER, GROK_SESSION_ID_OPTION, - GROK_USER_ID_OPTION, } from "../provider/grok-responses.js"; import { BIFROST_PROVIDER } from "../provider/bifrost-adapter.js"; import { isOllamaProviderId, ollamaOpenAIBaseURL } from "../provider/ollama.js"; @@ -117,11 +118,21 @@ export const KEYLESS_API_KEY = "keyless"; // ./source-credentials.ts), falling back to the keyless sentinel when no key // was configured. Every buildXSource below calls this so the vendored // credentialId auth model resolves the secret at send time. -function registerSourceSecret(id: string, apiKey: string | undefined): void { - registerSourceCredential( - id, - apiKey !== undefined && apiKey.length > 0 ? apiKey : KEYLESS_API_KEY, - ); +function registerSourceSecret( + id: string, + apiKey: string | undefined, + provenance?: SourceCredentialProvenance, + headers?: Readonly>, +): void { + const hasSecret = apiKey !== undefined && apiKey.length > 0; + registerSourceCredentialRecord(id, { + provenance: + provenance ?? (hasSecret ? { kind: "api-key" } : { kind: "keyless" }), + material: { + secret: hasSecret ? apiKey : KEYLESS_API_KEY, + ...(headers !== undefined ? { headers } : {}), + }, + }); } function applyPersistedOAuthDefaults( @@ -327,12 +338,12 @@ export type ProviderCatalogEntry = Omit< // Build the InferenceSource for a Codex OAuth profile. Routes to the // "codex-responses" adapter (the Codex backend speaks the Responses API, not -// Chat Completions) and carries the account id + a session id through -// providerOptions, where the adapter lifts them into request headers. The -// access token is registered in the credential cell under the source id; the -// harness resolves it as the bearer credential at send time. +// Chat Completions) and carries the session id through providerOptions. The +// access token and account id are registered together in the credential cell; +// the harness resolves both at send time. export function buildCodexSource(fields: { id: string; + profile: string; apiKey: string; model: string; sessionId: string; @@ -342,11 +353,16 @@ export function buildCodexSource(fields: { const providerOptions: Record = { [CODEX_SESSION_ID_OPTION]: fields.sessionId, }; - if (fields.accountId !== undefined) - providerOptions[CODEX_ACCOUNT_ID_OPTION] = fields.accountId; if (fields.reasoningEffort !== undefined) providerOptions["reasoning_effort"] = fields.reasoningEffort; - registerSourceSecret(fields.id, fields.apiKey); + registerSourceSecret( + fields.id, + fields.apiKey, + { kind: "oauth", provider: "codex", profile: fields.profile }, + fields.accountId !== undefined + ? { "chatgpt-account-id": fields.accountId } + : undefined, + ); return { id: fields.id, provider: CODEX_RESPONSES_PROVIDER, @@ -366,6 +382,7 @@ export function buildCodexSource(fields: { // thread routes to the same cache shard (store:false has no other signal). export function buildXaiSource(fields: { id: string; + profile: string; apiKey: string; model: string; sessionId: string; @@ -375,10 +392,14 @@ export function buildXaiSource(fields: { const providerOptions: Record = { [GROK_SESSION_ID_OPTION]: fields.sessionId, }; - if (userId !== undefined) providerOptions[GROK_USER_ID_OPTION] = userId; if (fields.reasoningEffort !== undefined) providerOptions["reasoning_effort"] = fields.reasoningEffort; - registerSourceSecret(fields.id, fields.apiKey); + registerSourceSecret( + fields.id, + fields.apiKey, + { kind: "oauth", provider: "xai", profile: fields.profile }, + userId !== undefined ? { "x-grok-user-id": userId } : undefined, + ); return { id: fields.id, provider: GROK_RESPONSES_PROVIDER, diff --git a/src/config/inference-sources.test.ts b/src/config/inference-sources.test.ts index 2a1219112..006a0cf90 100644 --- a/src/config/inference-sources.test.ts +++ b/src/config/inference-sources.test.ts @@ -13,8 +13,13 @@ import { } from "../provider/context-window.js"; import { createOpenAICompatibleAdapter } from "../provider/openai-compatible-adapter.js"; import { createInferenceDependencies } from "../provider/inference-dependencies.js"; -import { clearSourceCredentials } from "./source-credentials.js"; +import { + clearSourceCredentials, + readSourceCredentialRecord, +} from "./source-credentials.js"; import { OPENAI_RESPONSES_PROVIDER } from "../provider/openai-responses.js"; +import { CODEX_ACCOUNT_ID_OPTION } from "../provider/codex-responses.js"; +import { GROK_USER_ID_OPTION } from "../provider/grok-responses.js"; import { ZEN_MESSAGES_PROVIDER } from "../provider/anthropic-session-adapter.js"; import { firstClassProviderById } from "../../packages/first-class-providers/src/index.js"; import { @@ -85,6 +90,89 @@ afterEach(() => { clearSourceCredentials(); }); +describe("source credential provenance", () => { + test("OAuth provenance comes from catalog profile markers", () => { + const source = buildInferenceSourceForRef( + { provider: "codex/work", model: "gpt-5" }, + { + sessionId: "sess-oauth", + catalog: [ + { + name: "codex/work", + baseURL: "https://chatgpt.com/backend-api/codex", + apiKey: "oauth-token", + models: ["gpt-5"], + codexProfile: "work", + codexAccountId: "account-1", + }, + ], + }, + undefined, + ); + + if (source === null) throw new Error("expected Codex source"); + expect(source.defaults?.providerOptions).not.toHaveProperty( + CODEX_ACCOUNT_ID_OPTION, + ); + expect(readSourceCredentialRecord(source.credentialId).provenance).toEqual({ + kind: "oauth", + provider: "codex", + profile: "work", + }); + }); + + test("xAI identity lives only in mutable credential material", () => { + const source = buildInferenceSourceForRef( + { provider: "xai/work", model: "grok-code-fast-1" }, + { + sessionId: "sess-oauth", + catalog: [ + { + name: "xai/work", + baseURL: "https://api.x.ai/v1", + apiKey: "header.eyJzdWIiOiJ1c2VyLWEifQ.signature", + models: ["grok-code-fast-1"], + xaiProfile: "work", + }, + ], + }, + undefined, + ); + + if (source === null) throw new Error("expected xAI source"); + expect(source.defaults?.providerOptions).not.toHaveProperty( + GROK_USER_ID_OPTION, + ); + expect( + readSourceCredentialRecord(source.credentialId).material.headers, + ).toEqual({ "x-grok-user-id": "user-a" }); + }); + + test("namespaced API-key rows are not inferred as OAuth", () => { + const source = buildInferenceSourceForRef( + { provider: "codex/shadow", model: "relay-model" }, + { + sessionId: "sess-key", + catalog: [ + { + name: "codex/shadow", + baseURL: "https://relay.example/v1", + apiKey: "explicit-key", + models: ["relay-model"], + }, + ], + }, + undefined, + ); + + if (source === null) throw new Error("expected API-key source"); + expect(source.provider).toBe("openai-compatible"); + expect(readSourceCredentialRecord(source.credentialId).provenance).toEqual({ + kind: "api-key", + }); + }); +}); + describe("contextWindow / maxTokens split (CL-7784)", () => { test("setting contextWindow does not change the source output budget", () => { const source = buildInferenceSourceForRef( diff --git a/src/config/inference-sources.ts b/src/config/inference-sources.ts index a677091ae..e6a5fb476 100644 --- a/src/config/inference-sources.ts +++ b/src/config/inference-sources.ts @@ -93,6 +93,7 @@ export function buildInferenceSourceForRef( if (entry?.codexProfile !== undefined) { return buildCodexSource({ id: ref.provider, + profile: entry.codexProfile, apiKey: entry.apiKey ?? "", model: ref.model, sessionId: ctx.sessionId, @@ -105,6 +106,7 @@ export function buildInferenceSourceForRef( if (entry?.xaiProfile !== undefined) { return buildXaiSource({ id: ref.provider, + profile: entry.xaiProfile, apiKey: entry.apiKey ?? "", model: ref.model, sessionId: ctx.sessionId, diff --git a/src/config/oauth-stores-race.test.ts b/src/config/oauth-stores-race.test.ts new file mode 100644 index 000000000..c646f2c6e --- /dev/null +++ b/src/config/oauth-stores-race.test.ts @@ -0,0 +1,148 @@ +import { mkdtemp, rm } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { describe, expect, test } from "bun:test"; + +import { + loadCodexProfile, + loadXaiProfile, + removeCodexProfile, + removeXaiProfile, + saveCodexProfile, + saveXaiProfile, + updateCodexTokens, + updateXaiTokens, +} from "./oauth-stores.js"; + +describe("OAuth profile refresh compare-and-swap", () => { + test("stale Codex refresh cannot overwrite a newly saved profile", async () => { + const home = await mkdtemp(join(tmpdir(), "codex-profile-cas-")); + try { + await saveCodexProfile( + { + name: "work", + createdAt: 1, + tokens: { + access: "access-a", + refresh: "refresh-a", + expiresAt: 1, + accountId: "account-a", + }, + }, + home, + ); + await removeCodexProfile("work", home); + await saveCodexProfile( + { + name: "work", + createdAt: 2, + tokens: { + access: "access-b", + refresh: "refresh-b", + expiresAt: 2, + accountId: "account-b", + }, + }, + home, + ); + + const winner = await updateCodexTokens( + "work", + { + access: "stale-access", + refresh: "stale-refresh", + expiresAt: 3, + accountId: "stale-account", + }, + home, + "refresh-a", + ); + + expect(winner).toEqual({ + name: "work", + createdAt: 2, + tokens: { + access: "access-b", + refresh: "refresh-b", + expiresAt: 2, + accountId: "account-b", + }, + }); + expect(await loadCodexProfile("work", home)).toEqual({ + name: "work", + createdAt: 2, + tokens: { + access: "access-b", + refresh: "refresh-b", + expiresAt: 2, + accountId: "account-b", + }, + }); + } finally { + await rm(home, { recursive: true, force: true }); + } + }); + + test("stale xAI refresh cannot overwrite a newly saved profile", async () => { + const home = await mkdtemp(join(tmpdir(), "xai-profile-cas-")); + try { + await saveXaiProfile( + { + name: "work", + createdAt: 1, + tokens: { + access: "access-a", + refresh: "refresh-a", + expiresAt: 1, + }, + }, + home, + ); + await removeXaiProfile("work", home); + await saveXaiProfile( + { + name: "work", + createdAt: 2, + tokens: { + access: "access-b", + refresh: "refresh-b", + expiresAt: 2, + }, + }, + home, + ); + + const winner = await updateXaiTokens( + "work", + { + access: "stale-access", + refresh: "stale-refresh", + expiresAt: 3, + }, + home, + "refresh-a", + ); + + expect(winner).toEqual({ + name: "work", + createdAt: 2, + tokens: { + access: "access-b", + refresh: "refresh-b", + expiresAt: 2, + }, + }); + expect(await loadXaiProfile("work", home)).toEqual({ + name: "work", + createdAt: 2, + tokens: { + access: "access-b", + refresh: "refresh-b", + expiresAt: 2, + }, + }); + } finally { + await rm(home, { recursive: true, force: true }); + } + }); +}); diff --git a/src/config/source-credentials.ts b/src/config/source-credentials.ts index a3d0c17fb..f91ef198a 100644 --- a/src/config/source-credentials.ts +++ b/src/config/source-credentials.ts @@ -1,48 +1,67 @@ -// First-party credential cell backing the vendored inference auth model. -// -// Since the 1ad0104 re-vendor, `InferenceSource` carries no inline secret: -// it names a `credentialId` and every send resolves the secret through a -// `CredentialMaterialResolver` ("credential cell" in upstream terms — read -// `CredentialMaterialResolver`'s doc comment in the vendored -// `@intx/types`). This module is that cell for first-party API-key and -// OAuth access-token sources: each `buildXSource` in `./index.ts` registers -// the secret it was built with under the source id, and the inference entry -// points (`assemble-runtime`, subagent run, summarizer fallback) hand -// `readSourceCredentialMaterial` to the vendored trees as their resolver. -// -// Keyed by source id because ids are unique per live source within a -// process. The map lives at module scope so sources built in one layer -// (config) resolve in another (agent env, reactor options) without threading -// secrets through every intermediate shape. -import type { CredentialMaterialResolver } from "@intx/types"; - -const cell = new Map(); - -export function registerSourceCredential( +import type { + CredentialMaterial, + CredentialMaterialResolver, +} from "@intx/types"; + +export type SourceCredentialProvenance = + | { + readonly kind: "oauth"; + readonly provider: "codex" | "xai"; + readonly profile: string; + } + | { readonly kind: "api-key" } + | { readonly kind: "keyless" }; + +export interface SourceCredentialRecord { + readonly provenance: SourceCredentialProvenance; + readonly material: CredentialMaterial; +} + +const cell = new Map(); + +export function registerSourceCredentialRecord( credentialId: string, - secret: string, + record: SourceCredentialRecord, ): void { - cell.set(credentialId, secret); + cell.set(credentialId, record); } -/** The resolver handed to vendored inference calls. Fails closed. */ -export const readSourceCredentialMaterial: CredentialMaterialResolver = ( +export function rotateSourceCredentialMaterialIfCurrent( + credentialId: string, + expected: SourceCredentialRecord, + material: CredentialMaterial, +): boolean { + if (cell.get(credentialId) !== expected) return false; + cell.set(credentialId, { provenance: expected.provenance, material }); + return true; +} + +export function readSourceCredentialRecord( credentialId: string, -) => { - const secret = cell.get(credentialId); - if (secret === undefined) +): SourceCredentialRecord { + const record = cell.get(credentialId); + if (record === undefined) { throw new Error(`Unknown inference credential "${credentialId}".`); - return { secret }; -}; + } + return record; +} + +export function findSourceCredentialRecord( + credentialId: string, +): SourceCredentialRecord | undefined { + return cell.get(credentialId); +} + +export const readSourceCredentialMaterial: CredentialMaterialResolver = ( + credentialId: string, +) => readSourceCredentialRecord(credentialId).material; -/** Non-throwing read for "did the token change?" comparisons. */ export function peekSourceCredentialSecret( credentialId: string, ): string | undefined { - return cell.get(credentialId); + return cell.get(credentialId)?.material.secret; } -/** Test seam: empties the cell between cases. */ export function clearSourceCredentials(): void { cell.clear(); } diff --git a/src/diagnostic-sanitize.test.ts b/src/diagnostic-sanitize.test.ts new file mode 100644 index 000000000..80f34b214 --- /dev/null +++ b/src/diagnostic-sanitize.test.ts @@ -0,0 +1,40 @@ +import { describe, expect, test } from "bun:test"; + +import { + sanitizeDiagnosticText, + sanitizeDiagnosticValue, +} from "./diagnostic-sanitize.js"; + +describe("diagnostic sanitization", () => { + test("scrubs an opaque configured credential without dropping useful detail", () => { + const secret = "opaque phrase with spaces / punctuation?!"; + const diagnostic = + `provider rejected ${secret} while calling https://relay.example/v1 ` + + "with status 401"; + + const sanitized = sanitizeDiagnosticText(diagnostic, [secret]); + + expect(sanitized).not.toContain(secret); + expect(sanitized).toContain("provider rejected"); + expect(sanitized).toContain("https://relay.example/v1"); + expect(sanitized).toContain("status 401"); + }); + + test("recursively scrubs opaque credentials and terminal controls", () => { + const secret = "not-token-shaped"; + const sanitized = sanitizeDiagnosticValue( + { + error: `reflected ${secret}\u001b[31m`, + nested: [`still ${secret}`], + }, + [secret], + ); + + expect(JSON.stringify(sanitized)).not.toContain(secret); + expect(JSON.stringify(sanitized)).not.toContain("\\u001b"); + expect(sanitized).toEqual({ + error: "reflected [redacted: configured credential]", + nested: ["still [redacted: configured credential]"], + }); + }); +}); diff --git a/src/diagnostic-sanitize.ts b/src/diagnostic-sanitize.ts new file mode 100644 index 000000000..2055c1291 --- /dev/null +++ b/src/diagnostic-sanitize.ts @@ -0,0 +1,35 @@ +import { scrubSecretShapedContent } from "./plugins/tool-result-secret-scrub.js"; +import { stripTerminalControlSequences } from "./util/control-char-strip.js"; + +export function sanitizeDiagnosticText( + text: string, + configuredSecrets: readonly (string | undefined)[], +): string { + let sanitized = stripTerminalControlSequences(text); + for (const secret of configuredSecrets) { + if (secret !== undefined && secret.length > 0) + sanitized = sanitized + .split(secret) + .join("[redacted: configured credential]"); + } + return scrubSecretShapedContent(sanitized); +} + +export function sanitizeDiagnosticValue( + value: unknown, + configuredSecrets: readonly (string | undefined)[], +): unknown { + if (typeof value === "string") + return sanitizeDiagnosticText(value, configuredSecrets); + if (Array.isArray(value)) + return value.map((item) => + sanitizeDiagnosticValue(item, configuredSecrets), + ); + if (value !== null && typeof value === "object") { + const sanitized: Record = {}; + for (const [key, child] of Object.entries(value)) + sanitized[key] = sanitizeDiagnosticValue(child, configuredSecrets); + return sanitized; + } + return value; +} diff --git a/src/exec/dispose.ts b/src/exec/dispose.ts index 1f9be498b..bbded6a41 100644 --- a/src/exec/dispose.ts +++ b/src/exec/dispose.ts @@ -35,6 +35,7 @@ export function disposeExecRuntime(args: { agent: { close: () => Promise } | null; toolset: { dispose: () => Promise } | null; subAgentSessions: Pick | null; + sanitizeDiagnostic?: (message: string) => string; }): Promise { const key = args.toolset ?? args.agent ?? args.subAgentSessions; if (key !== null) { @@ -51,6 +52,7 @@ async function runExecDispose(args: { agent: { close: () => Promise } | null; toolset: { dispose: () => Promise } | null; subAgentSessions: Pick | null; + sanitizeDiagnostic?: (message: string) => string; }): Promise { const failures: unknown[] = []; if (args.toolset !== null) { @@ -58,7 +60,7 @@ async function runExecDispose(args: { await args.toolset.dispose(); } catch (err: unknown) { logger.debug("toolset.dispose during exec finally failed: {error}", { - error: formatCaughtError(err), + error: (args.sanitizeDiagnostic ?? String)(formatCaughtError(err)), }); failures.push(err); } @@ -73,7 +75,7 @@ async function runExecDispose(args: { await awaitCloseWithoutHidingLeftover(args.agent.close(), failures[0]); } catch (err: unknown) { logger.debug("agent.close during exec finally failed: {error}", { - error: formatCaughtError(err), + error: (args.sanitizeDiagnostic ?? String)(formatCaughtError(err)), }); failures.push(err); } diff --git a/src/exec/runner.test.ts b/src/exec/runner.test.ts index 8d1554026..8a13244ef 100644 --- a/src/exec/runner.test.ts +++ b/src/exec/runner.test.ts @@ -365,8 +365,8 @@ describe("exec credential failure surface", () => { ); }); - test("the credential failure message itself carries the re-login hint", () => { - expect(CREDENTIAL_FAILURE_USER_MESSAGE).toMatch(/log in again/i); + test("the credential failure message itself carries the /connect path", () => { + expect(CREDENTIAL_FAILURE_USER_MESSAGE).toContain("/connect"); }); test("a codex refresh lock failure keeps its own message with the lock path", async () => { diff --git a/src/exec/runner.ts b/src/exec/runner.ts index bbb3f2a39..4b5ae9696 100644 --- a/src/exec/runner.ts +++ b/src/exec/runner.ts @@ -10,14 +10,10 @@ import { toolWatchdogFromSettings, type MCPServerConfig, } from "../config/settings.js"; +import { isCodexProviderName } from "../config/codex-providers.js"; import { - codexProfileFromProviderName, - isCodexProviderName, -} from "../config/codex-providers.js"; -import { xaiProfileFromProviderName } from "../config/xai-providers.js"; -import { + findSourceCredentialRecord, peekSourceCredentialSecret, - registerSourceCredential, } from "../config/source-credentials.js"; import { formatDirectorSystemPrompt } from "../agent/directors/identity.js"; import { DIRECTOR_REGISTRY } from "../agent/directors/registry.js"; @@ -31,9 +27,7 @@ import { import { CodexRefreshLockError, codexAuthFailureDiagnostic, - getValidCodexToken, } from "../auth/codex/session.js"; -import { getValidXaiToken } from "../auth/xai/session.js"; import { type ActivatedToolTracker, type ToolAvailability, @@ -151,6 +145,10 @@ import { import type { ReactorEmittedEvent } from "@intx/inference"; import { setAgentSourceUnlessClosed } from "../tui/agent-source-sync.js"; import { ensureFreshInferenceSource } from "../subagent/refresh-inference-source.js"; +import { + sanitizeDiagnosticText, + sanitizeDiagnosticValue, +} from "../diagnostic-sanitize.js"; import { MAX_TOOL_APPROVAL_PAUSE_MS, getToolApprovalBudget, @@ -468,6 +466,13 @@ export async function runExec(config: Config): Promise { let providerFailureObserved = false; let providerError: InferenceErrorLike | undefined; let result: ExecResult | undefined; + let liveCredentialId: string | undefined; + const sanitizeExecDiagnostic = (text: string): string => + sanitizeDiagnosticText(text, [ + liveCredentialId === undefined + ? undefined + : peekSourceCredentialSecret(liveCredentialId), + ]); // Assigned once the advertised toolset exists (below); persist reads it live // so a snapshot taken before that point still writes, just without the field. const activatedToolsRef: { current?: ActivatedToolTracker } = {}; @@ -520,7 +525,9 @@ export async function runExec(config: Config): Promise { ? { lastCacheWriteAt: activeRunHandle.lastCacheWriteAt } : {}), ...(status !== "running" ? { finishedAt: Date.now() } : {}), - ...(extra?.error !== undefined ? { error: extra.error } : {}), + ...(extra?.error !== undefined + ? { error: sanitizeExecDiagnostic(extra.error) } + : {}), }; const write = status === "running" @@ -538,7 +545,7 @@ export async function runExec(config: Config): Promise { { sessionId, status, - error: formatCaughtError(err), + error: sanitizeExecDiagnostic(formatCaughtError(err)), }, ); }); @@ -554,7 +561,7 @@ export async function runExec(config: Config): Promise { // Pricing seed is optional for exec; continue without rates rather than fail the run. const inferenceDeps = await assembleInferenceBase((err: unknown) => { logger.debug("seedPricingMetadataFromCache failed: {error}", { - error: formatCaughtError(err), + error: sanitizeExecDiagnostic(formatCaughtError(err)), }); }); @@ -599,7 +606,7 @@ export async function runExec(config: Config): Promise { globalSettingsPath: config.globalSettingsPath, onError: (err: unknown) => { logger.warn("Failed to load local settings: {error}", { - error: formatCaughtError(err), + error: sanitizeExecDiagnostic(formatCaughtError(err)), }); }, }); @@ -744,7 +751,12 @@ export async function runExec(config: Config): Promise { }); toolset = agentToolset; setActiveDisposeHost(() => - disposeExecRuntime({ agent, toolset, subAgentSessions }), + disposeExecRuntime({ + agent, + toolset, + subAgentSessions, + sanitizeDiagnostic: sanitizeExecDiagnostic, + }), ); const systemPrompt = @@ -767,35 +779,25 @@ export async function runExec(config: Config): Promise { }) ).systemPrompt; - const initialCodexProfile = codexProfileFromProviderName( - config.providerName, - ); - const initialXaiProfile = xaiProfileFromProviderName(config.providerName); const initialBundle = resolveLiveSessionSources(config, sessionId); const liveSources = initialBundle.sources; const liveDefaultSource = initialBundle.defaultSource; const selectedSource = initialBundle.selected; + liveCredentialId = selectedSource.credentialId; + const initialProvenance = findSourceCredentialRecord( + selectedSource.credentialId, + )?.provenance; let liveSource: InferenceSource = selectedSource; // Refresh OAuth tokens before first inference when starting on codex/xai. - if (initialCodexProfile !== undefined) { - const { access } = await refreshSelectedProviderCredential(() => - getValidCodexToken(initialCodexProfile), + if (initialProvenance?.kind === "oauth") { + await refreshSelectedProviderCredential(() => + ensureFreshInferenceSource(liveSource, config.providers), ); - registerSourceCredential(liveSource.credentialId, access); + const access = peekSourceCredentialSecret(liveSource.credentialId); liveSubAgentProvider.current = { ...liveSubAgentProvider.current, - apiKey: access, - }; - } - if (initialXaiProfile !== undefined) { - const { access } = await refreshSelectedProviderCredential(() => - getValidXaiToken(initialXaiProfile), - ); - registerSourceCredential(liveSource.credentialId, access); - liveSubAgentProvider.current = { - ...liveSubAgentProvider.current, - apiKey: access, + ...(access !== undefined ? { apiKey: access } : {}), }; } @@ -995,7 +997,7 @@ export async function runExec(config: Config): Promise { handshake, ).catch((err: unknown) => { logger.warn("MCP connect failed: {error}", { - error: formatCaughtError(err), + error: sanitizeExecDiagnostic(formatCaughtError(err)), }); }); await awaitExecMcpThenResume(connecting, () => workflowHost.resume(), { @@ -1030,13 +1032,16 @@ export async function runExec(config: Config): Promise { // its partial output in partial.jsonl instead of vanishing. const sink = (event: ReactorEmittedEvent): void => { approvalAcceptance.observe(event); + const sanitizedEvent = sanitizeDiagnosticValue(event, [ + peekSourceCredentialSecret(liveSource.credentialId), + ]) as ReactorEmittedEvent; // Chat-director reactor events (replacing the former onTasksChange / // onActivateTools closures). Exec mode has no live task panel or task // stdout output today (unlike the TUI's chrome zone) — debug logging // is the closest match to how this mode already surfaces other // in-session state changes. handleChatDirectorEvent( - event, + sanitizedEvent, { onTasksChanged: (tasks) => { logger.debug("tasks updated: {tasks}", { @@ -1047,12 +1052,15 @@ export async function runExec(config: Config): Promise { }, (message, fields) => logger.debug(message, fields), ); - if (event.type === "inference.start" || event.type === "inference.done") { + if ( + sanitizedEvent.type === "inference.start" || + sanitizedEvent.type === "inference.done" + ) { providerFailureObserved = false; providerError = undefined; - } else if (event.type === "inference.error") { + } else if (sanitizedEvent.type === "inference.error") { providerFailureObserved = true; - const error = event.data.error; + const error = sanitizedEvent.data.error; providerError = { category: error.category, ...(error.message !== undefined ? { message: error.message } : {}), @@ -1063,18 +1071,18 @@ export async function runExec(config: Config): Promise { ? { providerId: error.providerId } : {}), }; - } else if (event.type === COMPACTION_CONTINUATION_EVENT) { + } else if (sanitizedEvent.type === COMPACTION_CONTINUATION_EVENT) { // Compaction governor self-delivers after compact so the loop re-enters. // Each emission is answered once: a replayed duplicate of an // already-answered emission is ignored instead of re-delivered. - if (continuationGate.shouldDeliver(event.seq)) { + if (continuationGate.shouldDeliver(sanitizedEvent.seq)) { currentAgent?.deliver(buildCompactionContinuationMessage()); } } - liveSink.sink(event); - cycleRecorder.handleEvent(event); - if (event.type === "inference.text.delta") { - const token = (event.data as { token?: string }).token; + liveSink.sink(sanitizedEvent); + cycleRecorder.handleEvent(sanitizedEvent); + if (sanitizedEvent.type === "inference.text.delta") { + const token = (sanitizedEvent.data as { token?: string }).token; if (typeof token === "string" && token.length > 0) { textChunks.push(token); output.write(token); @@ -1095,17 +1103,10 @@ export async function runExec(config: Config): Promise { let sinkStatus: ReturnType = "cancelled"; try { // Final OAuth refresh immediately before send (token may have aged during MCP). - if (initialCodexProfile !== undefined) { - const { access } = await getValidCodexToken(initialCodexProfile); - if (access !== peekSourceCredentialSecret(liveSource.credentialId)) { - registerSourceCredential(liveSource.credentialId, access); - setAgentSourceUnlessClosed(activeAgent, liveSource); - } - } - if (initialXaiProfile !== undefined) { - const { access } = await getValidXaiToken(initialXaiProfile); - if (access !== peekSourceCredentialSecret(liveSource.credentialId)) { - registerSourceCredential(liveSource.credentialId, access); + if (initialProvenance?.kind === "oauth") { + const before = peekSourceCredentialSecret(liveSource.credentialId); + await ensureFreshInferenceSource(liveSource, config.providers); + if (peekSourceCredentialSecret(liveSource.credentialId) !== before) { setAgentSourceUnlessClosed(activeAgent, liveSource); } } @@ -1141,7 +1142,7 @@ export async function runExec(config: Config): Promise { logger.debug( "agent.close during successful-send teardown failed: {error}", { - error: formatCaughtError(err), + error: sanitizeExecDiagnostic(formatCaughtError(err)), }, ); }); @@ -1149,7 +1150,7 @@ export async function runExec(config: Config): Promise { logger.debug( "stream drain during successful-send teardown failed: {error}", { - error: formatCaughtError(err), + error: sanitizeExecDiagnostic(formatCaughtError(err)), }, ); }); @@ -1166,7 +1167,7 @@ export async function runExec(config: Config): Promise { logger.debug( "agent.close during failed-send teardown failed: {error}", { - error: formatCaughtError(err), + error: sanitizeExecDiagnostic(formatCaughtError(err)), }, ); }); @@ -1174,7 +1175,7 @@ export async function runExec(config: Config): Promise { logger.debug( "stream drain during failed-send teardown failed: {error}", { - error: formatCaughtError(err), + error: sanitizeExecDiagnostic(formatCaughtError(err)), }, ); }); @@ -1210,7 +1211,7 @@ export async function runExec(config: Config): Promise { await hookManager.dispatchPostRun(runSummary).catch((err: unknown) => { // Post-run hooks are best-effort; keep the exec exit path intact but // surface the failure so operators can see hook/script problems. - const message = formatCaughtError(err); + const message = sanitizeExecDiagnostic(formatCaughtError(err)); logger.warn("dispatchPostRun failed: {error}", { error: message }); stderr.write(`Warning: post-run hook failed: ${message}\n`); }); @@ -1270,13 +1271,15 @@ export async function runExec(config: Config): Promise { }; return result; } catch (err) { - const diagnosticMessage = formatCaughtError(err); + const diagnosticMessage = sanitizeExecDiagnostic(formatCaughtError(err)); logger.error("exec failed: {error}", { error: diagnosticMessage }); - const userMessage = execUserFailureMessage( - config, - err, - providerFailureObserved, - providerError, + const userMessage = sanitizeExecDiagnostic( + execUserFailureMessage( + config, + err, + providerFailureObserved, + providerError, + ), ); stderr.write(`Error: ${userMessage}\n`); await persist("failed", { error: diagnosticMessage }); @@ -1302,9 +1305,14 @@ export async function runExec(config: Config): Promise { return result; } finally { try { - await disposeExecRuntime({ agent, toolset, subAgentSessions }); + await disposeExecRuntime({ + agent, + toolset, + subAgentSessions, + sanitizeDiagnostic: sanitizeExecDiagnostic, + }); } catch (err: unknown) { - const message = formatCaughtError(err); + const message = sanitizeExecDiagnostic(formatCaughtError(err)); logger.error("runtime dispose failed: {error}", { error: message }); stderr.write(`Error: runtime dispose failed: ${message}\n`); if (result !== undefined) { diff --git a/src/inference-error-message.test.ts b/src/inference-error-message.test.ts index 81e5b2f32..69f8d068b 100644 --- a/src/inference-error-message.test.ts +++ b/src/inference-error-message.test.ts @@ -86,17 +86,28 @@ describe("inferenceErrorMessage", () => { expect(line.toLowerCase()).not.toContain("usage limit reached"); }); - test("credential_failure tells the user to log in again", () => { + test("credential_failure tells the user to run /connect", () => { const line = inferenceErrorMessage({ category: "credential_failure", message: '{"error":{"code":401}}', }); - expect(line.toLowerCase()).not.toContain("re-authenticating"); - expect(line.toLowerCase()).toMatch(/log in again|sign in again/); + expect(line).toContain("/connect"); + expect(line.toLowerCase()).not.toMatch(/log in again|sign in again/); }); }); describe("terminalProviderFailureMessage", () => { + test("preserves full sanitized recovery URLs without a user-facing clamp", () => { + const recoveryURL = `https://auth.example/connect?state=${"s".repeat(260)}&profile=work`; + const message = terminalProviderFailureMessage("codex/work", { + category: "credential_failure", + message: `Reconnect with ${recoveryURL}; Authorization: Bearer secret-token-1234567890`, + }); + + expect(message).toContain(recoveryURL); + expect(message).not.toContain("secret-token-1234567890"); + }); + test("surfaces a retryable HTTP failure with safe retry guidance", () => { expect( terminalProviderFailureMessage( @@ -148,7 +159,7 @@ describe("terminalProviderFailureMessage", () => { ); }); - test("tells the user to log in again after a credential failure", () => { + test("tells the user to run /connect after a credential failure", () => { expect( terminalProviderFailureMessage("custom-provider", { category: "credential_failure", @@ -156,7 +167,7 @@ describe("terminalProviderFailureMessage", () => { statusCode: 401, }), ).toBe( - "custom-provider Provider failed (credential_failure): HTTP 401 Unauthorized. Authentication failed — log in again.", + "custom-provider Provider failed (credential_failure): HTTP 401 Unauthorized. Authentication failed — run /connect to reconnect the provider profile.", ); }); @@ -179,9 +190,9 @@ describe("terminalProviderFailureMessage", () => { expect(message).toContain('Codex profile "work"'); expect(message).toContain("Not Found"); expect(message).not.toContain("/model"); - // One re-login hint, not a stutter: the branded diagnostic dedups via - // the shared carriesCodexReLoginHint predicate. - expect(message.toLowerCase().match(/log in again/g)).toHaveLength(1); + expect(message).toContain("/connect"); + expect(message).toContain("Codex"); + expect(message).not.toMatch(/log in again|sign in again/i); }); test("terminal bare Codex 404 without an auth signal keeps switch-models guidance", () => { @@ -296,13 +307,14 @@ describe("terminalProviderFailureMessage", () => { expect(message).not.toContain("\u001b"); }); - test("bounds provider-controlled display text", () => { + test("preserves full provider diagnostics", () => { + const diagnostic = "x".repeat(1_000); const message = terminalProviderFailureMessage("custom-provider", { category: "fatal", - message: "x".repeat(1_000), + message: diagnostic, }); - expect(message).toContain(`${"x".repeat(239)}…`); - expect(message).not.toContain("x".repeat(241)); + expect(message).toContain(diagnostic); + expect(message).not.toContain("…"); }); }); diff --git a/src/inference-error-message.ts b/src/inference-error-message.ts index 83206c65d..152605a73 100644 --- a/src/inference-error-message.ts +++ b/src/inference-error-message.ts @@ -26,7 +26,7 @@ import { /** Committed auth death — do not claim a refresh is in flight. */ export const CREDENTIAL_FAILURE_USER_MESSAGE = - "Authentication failed — log in again."; + "Authentication failed — run /connect to reconnect the provider profile."; const FRIENDLY_BY_CATEGORY: Record = { credential_failure: CREDENTIAL_FAILURE_USER_MESSAGE, @@ -123,14 +123,13 @@ function codexUsageLimitLine(error: InferenceErrorLike): string | undefined { }); } -const TERMINAL_DIAGNOSTIC_MAX_CHARS = 240; const TERMINAL_PROVIDER_LABEL_MAX_CHARS = 80; -function safeDisplayText(text: string, maxChars: number): string { +function safeDisplayText(text: string, maxChars?: number): string { const oneLine = scrubSecretShapedContent(stripTerminalControlSequences(text)) .replace(/\s+/g, " ") .trim(); - return oneLine.length > maxChars + return maxChars !== undefined && oneLine.length > maxChars ? `${oneLine.slice(0, maxChars - 1)}…` : oneLine; } @@ -163,10 +162,7 @@ export function terminalProviderFailureMessage( ): string { const label = terminalProviderFailureLabel(providerId, displayLabel); const category = terminalProviderFailureCategory(error); - const message = safeDisplayText( - error.message ?? "", - TERMINAL_DIAGNOSTIC_MAX_CHARS, - ); + const message = safeDisplayText(error.message ?? ""); const diagnostic = message.length > 0 ? message : "inference error"; const diagnosticSentence = /[.!?]$/.test(diagnostic) ? diagnostic @@ -278,10 +274,7 @@ export function inferenceErrorMessage(error: InferenceErrorLike): string { if (codexLine !== undefined) return codexLine; } - const fallback = safeDisplayText( - error.message ?? "", - TERMINAL_DIAGNOSTIC_MAX_CHARS, - ); + const fallback = safeDisplayText(error.message ?? ""); return ( FRIENDLY_BY_CATEGORY[category] ?? (fallback.length > 0 ? fallback : "inference error") diff --git a/src/inference-gateway-error.ts b/src/inference-gateway-error.ts index 12c652831..b15b940ed 100644 --- a/src/inference-gateway-error.ts +++ b/src/inference-gateway-error.ts @@ -582,7 +582,7 @@ function looksLikeCodexModelDeprecation(error: InferenceErrorLike): boolean { * terminal-guidance dedup checks with it, so the two cannot drift. */ export function carriesCodexReLoginHint(text: string): boolean { - return /log in again|sign in again/i.test(text); + return /\/connect/i.test(text) && /codex profile/i.test(text); } /** Branded re-login line for a Codex credential 404, diagnostic appended. */ @@ -590,7 +590,7 @@ function formatCodexCredential404Message( profile: string, originalDiagnostic: string, ): string { - const branded = `Codex profile "${profile}" is not authorized. Log in again.`; + const branded = `Codex profile "${profile}" is not authorized. Run /connect, choose Codex, and reconnect profile "${profile}".`; const oneLine = originalDiagnostic.replace(/\s+/g, " ").trim(); if (oneLine.length === 0 || branded.includes(oneLine)) return branded; const clipped = oneLine.length > 200 ? `${oneLine.slice(0, 199)}…` : oneLine; diff --git a/src/subagent/refresh-inference-source.test.ts b/src/subagent/refresh-inference-source.test.ts index c7fbea0f6..f0e3b5a93 100644 --- a/src/subagent/refresh-inference-source.test.ts +++ b/src/subagent/refresh-inference-source.test.ts @@ -6,6 +6,8 @@ import type { InferenceSource } from "@intx/types/runtime"; import { clearSourceCredentials, peekSourceCredentialSecret, + readSourceCredentialMaterial, + registerSourceCredentialRecord, } from "../config/source-credentials.js"; const baseSource = (id: string): InferenceSource => ({ @@ -30,6 +32,10 @@ describe("refresh-inference-source", () => { const { ensureFreshInferenceSource } = await import("./refresh-inference-source.js"); const source = baseSource("codex/default"); + registerSourceCredentialRecord(source.credentialId, { + provenance: { kind: "oauth", provider: "codex", profile: "default" }, + material: { secret: "stale-codex-token" }, + }); const out = await ensureFreshInferenceSource(source, []); expect(out).toBe(source); expect(peekSourceCredentialSecret(source.credentialId)).toBe( @@ -37,6 +43,96 @@ describe("refresh-inference-source", () => { ); }); + test("refresh replaces Codex identity and removes a missing identity", async () => { + const refresh = spyOn(codexSession, "getValidCodexToken"); + refresh.mockResolvedValueOnce({ + access: "token-b", + accountId: "account-b", + }); + refresh.mockResolvedValueOnce({ access: "token-c" }); + const { ensureFreshInferenceSource } = + await import("./refresh-inference-source.js"); + const source = baseSource("codex/work"); + registerSourceCredentialRecord(source.credentialId, { + provenance: { kind: "oauth", provider: "codex", profile: "work" }, + material: { + secret: "token-a", + headers: { "chatgpt-account-id": "account-a" }, + }, + }); + + await ensureFreshInferenceSource(source, []); + expect(readSourceCredentialMaterial(source.credentialId)).toEqual({ + secret: "token-b", + headers: { "chatgpt-account-id": "account-b" }, + }); + + await ensureFreshInferenceSource(source, []); + expect(readSourceCredentialMaterial(source.credentialId)).toEqual({ + secret: "token-c", + }); + }); + + test("refresh replaces xAI identity and removes a missing identity", async () => { + const accessWithUser = "header.eyJzdWIiOiJ1c2VyLWIifQ.signature"; + const refresh = spyOn(xaiSession, "getValidXaiToken"); + refresh.mockResolvedValueOnce({ access: accessWithUser }); + refresh.mockResolvedValueOnce({ access: "opaque-token-without-user" }); + const { ensureFreshInferenceSource } = + await import("./refresh-inference-source.js"); + const source = baseSource("xai/work"); + registerSourceCredentialRecord(source.credentialId, { + provenance: { kind: "oauth", provider: "xai", profile: "work" }, + material: { + secret: "token-a", + headers: { "x-grok-user-id": "user-a" }, + }, + }); + + await ensureFreshInferenceSource(source, []); + expect(readSourceCredentialMaterial(source.credentialId)).toEqual({ + secret: accessWithUser, + headers: { "x-grok-user-id": "user-b" }, + }); + + await ensureFreshInferenceSource(source, []); + expect(readSourceCredentialMaterial(source.credentialId)).toEqual({ + secret: "opaque-token-without-user", + }); + }); + + test("deferred refresh cannot overwrite a newer credential registration", async () => { + let resolveRefresh!: (value: { access: string; accountId: string }) => void; + spyOn(codexSession, "getValidCodexToken").mockReturnValue( + new Promise((resolve) => { + resolveRefresh = resolve; + }), + ); + const { ensureFreshInferenceSource } = + await import("./refresh-inference-source.js"); + const source = baseSource("codex/work"); + registerSourceCredentialRecord(source.credentialId, { + provenance: { kind: "oauth", provider: "codex", profile: "work" }, + material: { secret: "token-a" }, + }); + + const pending = ensureFreshInferenceSource(source, []); + registerSourceCredentialRecord(source.credentialId, { + provenance: { kind: "oauth", provider: "codex", profile: "replacement" }, + material: { + secret: "token-b", + headers: { "chatgpt-account-id": "account-b" }, + }, + }); + resolveRefresh({ access: "stale-token", accountId: "stale-account" }); + await pending; + + expect(readSourceCredentialMaterial(source.credentialId)).toEqual({ + secret: "token-b", + headers: { "chatgpt-account-id": "account-b" }, + }); + }); + test("refreshInferenceSourceBundle refreshes each leg", async () => { const { refreshInferenceSourceBundle } = await import("./refresh-inference-source.js"); diff --git a/src/subagent/refresh-inference-source.ts b/src/subagent/refresh-inference-source.ts index 9af1fe5d1..c0ed60043 100644 --- a/src/subagent/refresh-inference-source.ts +++ b/src/subagent/refresh-inference-source.ts @@ -1,33 +1,13 @@ import type { InferenceSource } from "@intx/types/runtime"; -import { getValidCodexToken } from "../auth/codex/session.js"; -import { getValidXaiToken } from "../auth/xai/session.js"; +import { refreshSourceCredentialByProvenance } from "../auth/refresh-source-credential.js"; import type { ProviderCatalogEntry } from "../config/index.js"; -import { registerSourceCredential } from "../config/source-credentials.js"; -import { codexProfileFromProviderName } from "../config/codex-providers.js"; -import { xaiProfileFromProviderName } from "../config/xai-providers.js"; -// Sub-agents run their own reactor loop and do not inherit the TUI runner's -// refreshCodexBeforeSend hook. Refresh OAuth access tokens immediately before -// the first inference call so stale catalog snapshots do not surface as 401s. export async function ensureFreshInferenceSource( source: InferenceSource, - catalog: readonly ProviderCatalogEntry[] | undefined, + _catalog: readonly ProviderCatalogEntry[] | undefined, ): Promise { - const entry = catalog?.find((e) => e.name === source.id); - const codexProfile = - entry?.codexProfile ?? codexProfileFromProviderName(source.id); - if (codexProfile !== undefined) { - const { access } = await getValidCodexToken(codexProfile); - registerSourceCredential(source.credentialId, access); - return source; - } - const xaiProfile = entry?.xaiProfile ?? xaiProfileFromProviderName(source.id); - if (xaiProfile !== undefined) { - const { access } = await getValidXaiToken(xaiProfile); - registerSourceCredential(source.credentialId, access); - return source; - } + await refreshSourceCredentialByProvenance(source.credentialId); return source; } diff --git a/src/subagent/run-resolved-provider-failure.test.ts b/src/subagent/run-resolved-provider-failure.test.ts index a47c2d8e9..b2d52b5c7 100644 --- a/src/subagent/run-resolved-provider-failure.test.ts +++ b/src/subagent/run-resolved-provider-failure.test.ts @@ -23,15 +23,15 @@ import { unlimitedAdmissionQueue } from "./admission.js"; import { createSubAgentSessionStore } from "./session-store.js"; import type { RunSubAgentParams, RunSubAgentResult } from "./types.js"; -const RAW_DIAGNOSTIC = - "\u001b[31mPOST https://provider.invalid returned\n secret response body\u001b[0m"; -const NORMALIZED_DIAGNOSTIC = - "POST https://provider.invalid returned secret response body"; +const OPAQUE_SECRET = "opaque credential with spaces?!"; +const RAW_DIAGNOSTIC = `\u001b[31mPOST https://provider.invalid returned\n credential ${OPAQUE_SECRET} in response body\u001b[0m`; +const NORMALIZED_DIAGNOSTIC = `POST https://provider.invalid returned credential ${OPAQUE_SECRET} in response body`; const SAFE_MESSAGE = 'test-provider Provider failed (fatal). Try again or switch models with "/model".'; const provider = { providerName: "test-provider", baseURL: "http://localhost", + apiKey: OPAQUE_SECRET, model: "test-model", }; const testPermissionGate = createPermissionGate({ @@ -286,13 +286,18 @@ describe("resolved sub-agent provider failures", () => { expect((caught as ResolvedProviderFailureError).category).toBe("fatal"); expect(JSON.stringify(caught)).not.toContain(RAW_DIAGNOSTIC); expect(JSON.stringify(caught)).not.toContain(NORMALIZED_DIAGNOSTIC); - expect( - observed.some( - (event) => - event.type === "inference.error" && - event.data.error.message === RAW_DIAGNOSTIC, - ), - ).toBe(true); + const observedError = observed.find( + (event) => event.type === "inference.error", + ); + if (observedError?.type !== "inference.error") + throw new Error("expected sanitized inference.error"); + expect(observedError.data.error.message).toContain( + "POST https://provider.invalid returned", + ); + expect(observedError.data.error.message).toContain("in response body"); + expect(observedError.data.error.message).not.toContain(OPAQUE_SECRET); + expect(observedError.data.error.message).not.toContain("\u001b"); + expect(JSON.stringify(observed)).not.toContain(RAW_DIAGNOSTIC); }); test("split spawn_agent and wait_agents return only the safe message", async () => { diff --git a/src/subagent/run.ts b/src/subagent/run.ts index 1fb518a27..47bcdc6a2 100644 --- a/src/subagent/run.ts +++ b/src/subagent/run.ts @@ -46,6 +46,7 @@ import { buildSubagentSources, } from "../config/inference-sources.js"; import { readSourceCredentialMaterial } from "../config/source-credentials.js"; +import { sanitizeDiagnosticValue } from "../diagnostic-sanitize.js"; import { assembleInferenceBase } from "../session/assemble-runtime.js"; import { advertiseShellGuardTimeout } from "../plugins/shell-guard-plugin.js"; import { advertiseEditFileLineRange } from "../plugins/edit-file-line-range.js"; @@ -1238,6 +1239,11 @@ async function runSubAgentInner( params.catalog, params.settings, ); + const workerSource = + bundle.sources.find((source) => source.id === bundle.defaultSource) ?? + bundle.sources[0]; + if (workerSource === undefined) + throw new Error("sub-agent source bundle is empty"); agent = await createAgentWithLiveToolDispatch(def, { sources: bundle.sources, defaultSource: bundle.defaultSource, @@ -1304,8 +1310,11 @@ async function runSubAgentInner( // turn boundary has completed yet to carry it. const cycleRecorder = createCycleTextRecorder(() => workdir); const runSettlement = createRunEventSettlement(); - const streamSink = (event: ReactorEmittedEvent): void => { - runSettlement.handleEvent(event); + const streamSink = (rawEvent: ReactorEmittedEvent): void => { + runSettlement.handleEvent(rawEvent); + const event = sanitizeDiagnosticValue(rawEvent, [ + readSourceCredentialMaterial(workerSource.credentialId).secret, + ]) as ReactorEmittedEvent; const name = subAgentToolName(event); if (name !== null) { toolNamesUsed.push(name); diff --git a/src/tui/chrome-state-turn.test.ts b/src/tui/chrome-state-turn.test.ts index 8fda14bfd..85806ce80 100644 --- a/src/tui/chrome-state-turn.test.ts +++ b/src/tui/chrome-state-turn.test.ts @@ -335,8 +335,12 @@ describe("sendFailureText", () => { }); test("a classified credential_failure line is not rewritten as generic other", () => { - expect(sendFailureText("Authentication failed — log in again.")).toBe( - "Authentication failed — log in again.", + expect( + sendFailureText( + "Authentication failed — run /connect to reconnect the provider profile.", + ), + ).toBe( + "Authentication failed — run /connect to reconnect the provider profile.", ); }); }); diff --git a/src/tui/commands/built-in.test.ts b/src/tui/commands/built-in.test.ts index 5aafa1aa2..8fb6ce9b9 100644 --- a/src/tui/commands/built-in.test.ts +++ b/src/tui/commands/built-in.test.ts @@ -8,7 +8,11 @@ import { globalSettingsPath } from "../../config/settings.js"; import { createCommandLayer } from "../runner/commands.js"; import { createTUISettingsWriters } from "../runner/settings-writers.js"; import type { RunnerServices, RunnerState } from "../runner/state.js"; -import { getCommand } from "./registry.js"; +import { getCommand, listCommands } from "./registry.js"; +import { + commandItemsFromRegistry, + filterPaletteCommands, +} from "../command-catalog.js"; import type { CommandContext } from "./registry.js"; import { registerBuiltInCommands } from "./built-in.js"; import { buildCostSummary } from "../../cost/cost-summary.js"; @@ -63,6 +67,15 @@ describe("/connect command", () => { overlay: "add-provider", }); }); + + it("is discoverable by auth recovery terms", () => { + const catalog = commandItemsFromRegistry(listCommands()); + for (const query of ["auth", "login", "reauth", "credential"]) { + expect( + filterPaletteCommands(query, catalog).map((item) => item.id), + ).toContain("connect"); + } + }); }); describe("MCP commands", () => { diff --git a/src/tui/commands/built-in.ts b/src/tui/commands/built-in.ts index 4341bee0e..9ae3cf0cc 100644 --- a/src/tui/commands/built-in.ts +++ b/src/tui/commands/built-in.ts @@ -59,7 +59,8 @@ export function registerBuiltInCommands(): void { // standalone /login; OAuth sign-in is still reached only through this flow. registerCommand({ name: "connect", - description: "Add a provider account", + description: + "Connect or reauthenticate a provider account (auth, login, credentials)", handler: () => ({ type: "overlay", overlay: "add-provider" }), }); diff --git a/src/tui/model-catalog.test.ts b/src/tui/model-catalog.test.ts index 66befce59..fc431ac16 100644 --- a/src/tui/model-catalog.test.ts +++ b/src/tui/model-catalog.test.ts @@ -4,6 +4,7 @@ import { buildModelsFirstCatalog, describeModelCatalogOption, modelOptionId, + modelOptionRef, type ModelCatalogProvider, } from "./model-catalog"; @@ -14,9 +15,9 @@ describe("buildModelCatalog", () => { { name: "openai", models: ["gpt-5.6"] }, ]); expect(options).toEqual([ - { id: "xai:grok-4", label: "grok-4 * [xAI]" }, - { id: "xai:grok-3", label: "grok-3 * [xAI]" }, - { id: "openai:gpt-5.6", label: "gpt-5.6 * [openai]" }, + { id: modelOptionId("xai", "grok-4"), label: "grok-4 * [xAI]" }, + { id: modelOptionId("xai", "grok-3"), label: "grok-3 * [xAI]" }, + { id: modelOptionId("openai", "gpt-5.6"), label: "gpt-5.6 * [openai]" }, ]); }); @@ -26,9 +27,12 @@ describe("buildModelCatalog", () => { zen: { models: ["claude-sonnet-4-5"], label: "Zen" }, }); expect(options).toEqual([ - { id: "fp:fp-small", label: "fp-small * [fp]" }, - { id: "fp:fp-large", label: "fp-large * [fp]" }, - { id: "zen:claude-sonnet-4-5", label: "claude-sonnet-4-5 * [Zen]" }, + { id: modelOptionId("fp", "fp-small"), label: "fp-small * [fp]" }, + { id: modelOptionId("fp", "fp-large"), label: "fp-large * [fp]" }, + { + id: modelOptionId("zen", "claude-sonnet-4-5"), + label: "claude-sonnet-4-5 * [Zen]", + }, ]); }); @@ -38,14 +42,18 @@ describe("buildModelCatalog", () => { { name: "empty", models: [] }, { name: "blank", models: [" ", "keep"] }, ]), - ).toEqual([{ id: "blank:keep", label: "keep * [blank]" }]); + ).toEqual([ + { id: modelOptionId("blank", "keep"), label: "keep * [blank]" }, + ]); }); test("dedupes by provider:model id", () => { const options = buildModelCatalog([ { name: "xai", models: ["grok-4", "grok-4"] }, ]); - expect(options).toEqual([{ id: "xai:grok-4", label: "grok-4 * [xai]" }]); + expect(options).toEqual([ + { id: modelOptionId("xai", "grok-4"), label: "grok-4 * [xai]" }, + ]); }); test("empty input yields empty catalog", () => { @@ -55,8 +63,41 @@ describe("buildModelCatalog", () => { }); describe("modelOptionId", () => { - test("provider:model", () => { - expect(modelOptionId("xai", "grok-4")).toBe("xai:grok-4"); + test("round-trips representative legal identities", () => { + const values = [ + "plain", + "leading[bracket", + "colon:value", + 'quote"value', + '["encoded","looking"]', + ]; + for (const provider of values) { + for (const model of values) { + expect(modelOptionRef(modelOptionId(provider, model))).toEqual({ + provider, + model, + }); + } + } + }); + + test("is unique across provider and model domains", () => { + const pairs = [ + ["a:b", "c"], + ["a", "b:c"], + ['["a","b"]', "c"], + ["a", '["b","c"]'], + ["[a", 'b:c"'], + ] as const; + expect( + new Set(pairs.map(([provider, model]) => modelOptionId(provider, model))) + .size, + ).toBe(pairs.length); + }); + + test("rejects malformed and non-canonical identities", () => { + for (const id of ["", "a:b", "[]", '["a"]', '["a","b","c"]']) + expect(modelOptionRef(id)).toBeNull(); }); }); @@ -89,10 +130,10 @@ describe("buildModelsFirstCatalog", () => { }); expect(list.map((r) => `${r.section}:${r.id}`)).toEqual([ - "recent:zen:claude-sonnet-4-5", - "favorites:xai:grok-4", - "provider:xai:grok-3", - "provider:zen:kimi-k2.7-code", + `recent:${modelOptionId("zen", "claude-sonnet-4-5")}`, + `favorites:${modelOptionId("xai", "grok-4")}`, + `provider:${modelOptionId("xai", "grok-3")}`, + `provider:${modelOptionId("zen", "kimi-k2.7-code")}`, ]); }); @@ -107,7 +148,7 @@ describe("buildModelsFirstCatalog", () => { }); expect(list.filter((r) => r.section === "recent").map((r) => r.id)).toEqual( - ["xai:grok-4"], + [modelOptionId("xai", "grok-4")], ); }); @@ -118,7 +159,9 @@ describe("buildModelsFirstCatalog", () => { favorites: [{ provider: "xai", model: "grok-4" }], }); - expect(list.filter((r) => r.id === "xai:grok-4")).toHaveLength(1); + expect( + list.filter((r) => r.id === modelOptionId("xai", "grok-4")), + ).toHaveLength(1); expect(list[0]?.section).toBe("recent"); }); @@ -172,7 +215,9 @@ describe("buildModelsFirstCatalog", () => { expect(recent?.warning).toMatch(/Go model on Zen path/); expect(recent?.label).not.toContain("Go model on Zen path"); - const goRow = list.find((r) => r.id === "opencode-go:kimi-k2.7-code"); + const goRow = list.find( + (r) => r.id === modelOptionId("opencode-go", "kimi-k2.7-code"), + ); expect(goRow?.warning).toBeUndefined(); }); @@ -183,7 +228,9 @@ describe("buildModelsFirstCatalog", () => { favorites: [], }); - const row = list.find((r) => r.id === "zen:kimi-k2.7-code"); + const row = list.find( + (r) => r.id === modelOptionId("zen", "kimi-k2.7-code"), + ); expect(row?.warning).toMatch(/Go model on Zen path/); }); @@ -201,7 +248,7 @@ describe("describeModelCatalogOption", () => { test("surfaces the Go-on-Zen billing warning as a consequence-toned impact, not the label", () => { const description = describeModelCatalogOption( { - id: "zen:kimi-k2.7-code", + id: modelOptionId("zen", "kimi-k2.7-code"), label: "kimi-k2.7-code * [OpenCode Zen]", warning: "Go model on Zen path", }, @@ -213,7 +260,7 @@ describe("describeModelCatalogOption", () => { test("reports pricing as unknown rather than inventing a number", () => { const description = describeModelCatalogOption( - { id: "xai:grok-4", label: "grok-4 * [xAI]" }, + { id: modelOptionId("xai", "grok-4"), label: "grok-4 * [xAI]" }, { pricing: null }, ); expect(description?.impact).toMatch(/pricing unknown/i); @@ -224,7 +271,7 @@ describe("describeModelCatalogOption", () => { // misreads as metered billing with a missing rate. const description = describeModelCatalogOption( { - id: "codex/default:gpt-5.1-codex-max", + id: modelOptionId("codex/default", "gpt-5.1-codex-max"), label: "gpt-5.1-codex-max * [Codex default]", }, { pricing: null }, @@ -236,7 +283,7 @@ describe("describeModelCatalogOption", () => { test("connected Grok rows state plan billing plainly instead of unknown pricing (CL-5606)", () => { const description = describeModelCatalogOption( { - id: "xai/work:grok-4", + id: modelOptionId("xai/work", "grok-4"), label: "grok-4 * [xAI work]", }, { pricing: null }, diff --git a/src/tui/model-catalog.ts b/src/tui/model-catalog.ts index 44fac0a33..80e997ca3 100644 --- a/src/tui/model-catalog.ts +++ b/src/tui/model-catalog.ts @@ -6,9 +6,10 @@ * the Go-on-Zen billing predicate are also plain data — callers own settings * and config loading. * - * Identity is `provider:model` (matches runner active-model string). + * Identity is opaque and collision-free across provider/model pairs. */ +import { type } from "arktype"; import { isGoModelOnZenPath as defaultIsGoModelOnZenPath } from "../provider/billing-product.js"; import { getActivePricingCache } from "../cost/cost-visibility.js"; import { @@ -99,9 +100,32 @@ export function buildModelCatalog( return out; } -/** Stable id for a provider+model pair (`provider:model`). */ +const ModelOptionIdentity = type(["string", "string"]); + +const MODEL_OPTION_ID_PREFIX = "model:"; + +/** Stable opaque id for one exact provider/model pair. */ export function modelOptionId(provider: string, model: string): string { - return `${provider}:${model}`; + return `${MODEL_OPTION_ID_PREFIX}${JSON.stringify([provider, model])}`; +} + +export function modelOptionRef(id: string): ModelCatalogRef | null { + if (!id.startsWith(MODEL_OPTION_ID_PREFIX)) return null; + try { + const parsed = ModelOptionIdentity( + JSON.parse(id.slice(MODEL_OPTION_ID_PREFIX.length)), + ); + if ( + parsed instanceof type.errors || + parsed[0].length === 0 || + parsed[1].length === 0 || + modelOptionId(parsed[0], parsed[1]) !== id + ) + return null; + return { provider: parsed[0], model: parsed[1] }; + } catch { + return null; + } } /** Picker row: `model * [providerLabel]`. */ @@ -307,7 +331,9 @@ export function describeModelCatalogOption( readonly pricing?: PricingCache | null; }, ): ItemDescription | null { - const model = option.id.slice(option.id.indexOf(":") + 1); + const identity = modelOptionRef(option.id); + const provider = identity?.provider ?? option.id; + const model = identity?.model ?? option.id; const pricing = args?.pricing !== undefined ? args.pricing : getActivePricingCache(); @@ -322,13 +348,7 @@ export function describeModelCatalogOption( return { what: whatLine(model), - impact: pricingImpact( - pricing, - // Exact provider parse: slice(0, indexOf(":")) drops the last - // character of a colon-less id (indexOf returns -1). - option.id.split(":")[0] ?? option.id, - model, - ), + impact: pricingImpact(pricing, provider, model), tone: "plain", }; } diff --git a/src/tui/product-host.test.ts b/src/tui/product-host.test.ts index 48a737a2b..bc28a665a 100644 --- a/src/tui/product-host.test.ts +++ b/src/tui/product-host.test.ts @@ -442,7 +442,9 @@ describe("flat type-to-filter model picker", () => { expect(grokIndex).toBeGreaterThanOrEqual(0); moveOverlaySelection(host.shell, grokIndex); acceptOverlaySelection(host.shell); - expect(selected).toEqual(["xai/thegreataxios:grok-4.5"]); + expect(selected).toEqual([ + modelOptionId("xai/thegreataxios", "grok-4.5"), + ]); } finally { host.dispose(); harness.destroy(); @@ -459,7 +461,9 @@ describe("flat type-to-filter model picker", () => { expect(grokIndex).toBeGreaterThanOrEqual(0); moveOverlaySelection(host.shell, grokIndex); acceptOverlaySelection(host.shell); - expect(selected).toEqual(["xai/thegreataxios:grok-4.5"]); + expect(selected).toEqual([ + modelOptionId("xai/thegreataxios", "grok-4.5"), + ]); } finally { host.dispose(); harness.destroy(); @@ -597,7 +601,9 @@ describe("flat type-to-filter model picker", () => { await harness.renderOnce(); // Accept whatever is focused after filter (should be the sole match). acceptOverlaySelection(host.shell); - expect(selected).toEqual(["xai/thegreataxios:grok-4.5"]); + expect(selected).toEqual([ + modelOptionId("xai/thegreataxios", "grok-4.5"), + ]); } finally { host.dispose(); harness.destroy(); @@ -642,7 +648,7 @@ describe("flat type-to-filter model picker", () => { host.openModels?.(); await harness.renderOnce(); expect(runOverlayAction(host.shell, altD)).toBe(true); - expect(defaults).toEqual(["codex/abk-labs:gpt-5.5"]); + expect(defaults).toEqual([modelOptionId("codex/abk-labs", "gpt-5.5")]); expect(host.shell.overlayKind).toBe("model_picker"); } finally { host.dispose(); @@ -667,7 +673,7 @@ describe("flat type-to-filter model picker", () => { } as KeyEvent; expect(handleListFilterKey(host.shell, composed)).toBe(false); expect(runOverlayAction(host.shell, composed)).toBe(true); - expect(defaults).toEqual(["codex/abk-labs:gpt-5.5"]); + expect(defaults).toEqual([modelOptionId("codex/abk-labs", "gpt-5.5")]); expect(host.shell.overlayItems).not.toEqual(["(no matches)"]); } finally { host.dispose(); @@ -1240,7 +1246,7 @@ describe("flat type-to-filter model picker", () => { test("openModels(focusId) preselects the given row instead of the top of the list", async () => { const { harness, host } = await mountPicker(); try { - host.openModels?.("codex/abk-labs:gpt-5.6-sol"); + host.openModels?.(modelOptionId("codex/abk-labs", "gpt-5.6-sol")); await harness.renderOnce(); const idx = host.shell.overlayItems.findIndex((label) => label.includes("gpt-5.6-sol"), @@ -1263,8 +1269,14 @@ describe("flat type-to-filter model picker", () => { ).toBe(false); host.setModels?.([ - { id: "codex/abk-labs:gpt-5.5", label: "gpt-5.5 * [codex/abk-labs]" }, - { id: "opencode-go:live-1", label: "live-1 * [opencode-go]" }, + { + id: modelOptionId("codex/abk-labs", "gpt-5.5"), + label: "gpt-5.5 * [codex/abk-labs]", + }, + { + id: modelOptionId("opencode-go", "live-1"), + label: "live-1 * [opencode-go]", + }, ]); await harness.renderOnce(); @@ -1291,12 +1303,18 @@ describe("flat type-to-filter model picker", () => { expect(host.shell.overlayList?.activeIndex).toBe(grokIndex); host.setModels?.([ - { id: "opencode-go:live-1", label: "live-1 * [opencode-go]" }, { - id: "xai/thegreataxios:grok-4.5", + id: modelOptionId("opencode-go", "live-1"), + label: "live-1 * [opencode-go]", + }, + { + id: modelOptionId("xai/thegreataxios", "grok-4.5"), label: "grok-4.5 * [xai/thegreataxios]", }, - { id: "opencode-go:live-2", label: "live-2 * [opencode-go]" }, + { + id: modelOptionId("opencode-go", "live-2"), + label: "live-2 * [opencode-go]", + }, ]); await harness.renderOnce(); @@ -1327,7 +1345,10 @@ describe("flat type-to-filter model picker", () => { expect(host.shell.overlayKind).toBe("add_provider"); host.setModels?.([ - { id: "opencode-go:live-1", label: "live-1 * [opencode-go]" }, + { + id: modelOptionId("opencode-go", "live-1"), + label: "live-1 * [opencode-go]", + }, ]); await harness.renderOnce(); @@ -1354,11 +1375,17 @@ describe("flat type-to-filter model picker", () => { host.setModels?.([ { - id: "xai/thegreataxios:grok-4.5", + id: modelOptionId("xai/thegreataxios", "grok-4.5"), label: "grok-4.5 * [xai/thegreataxios]", }, - { id: "opencode-go:grok-live", label: "grok-live * [opencode-go]" }, - { id: "opencode-go:live-1", label: "live-1 * [opencode-go]" }, + { + id: modelOptionId("opencode-go", "grok-live"), + label: "grok-live * [opencode-go]", + }, + { + id: modelOptionId("opencode-go", "live-1"), + label: "live-1 * [opencode-go]", + }, ]); await harness.renderOnce(); diff --git a/src/tui/product-host.ts b/src/tui/product-host.ts index e24882440..9a73e9700 100644 --- a/src/tui/product-host.ts +++ b/src/tui/product-host.ts @@ -135,7 +135,7 @@ export interface ProductHostConfig { /** Model/provider rows for the picker (id applied on select). */ readonly models?: readonly ProductHostModelOption[]; /** - * Row id (`provider:model`) of the model the session is actually running, + * Opaque model option id of the model the session is actually running, * read live on every picker open so it tracks selections made outside the * picker (e.g. `defaultProvider` at startup). Marks that row "(current)" * instead of guessing from the recents list. diff --git a/src/tui/provider-setup.test.ts b/src/tui/provider-setup.test.ts index 1513244de..8914c3196 100644 --- a/src/tui/provider-setup.test.ts +++ b/src/tui/provider-setup.test.ts @@ -16,6 +16,7 @@ import { saveLocalSettings, } from "../config/settings.js"; import { createHarness as createRawHarness, type Harness } from "./harness.js"; +import { modelOptionId } from "./model-catalog.js"; import { addProviderSelectorChoices, connectedAccountCount, @@ -333,9 +334,13 @@ describe("provider setup pure helpers", () => { expect(openai).toBeDefined(); if (openai === undefined) return; const rows = modelChoiceRows(openai); - expect(rows.map((r) => r.id)).toContain(`openai:${openai.defaultModel}`); + expect(rows.map((r) => r.id)).toContain( + modelOptionId("openai", openai.defaultModel), + ); expect(rows.at(-1)?.id).toBe(TYPE_MODEL_ID); - expect(modelFromRowId("openai", "openai:gpt-5.4")).toBe("gpt-5.4"); + expect(modelFromRowId("openai", modelOptionId("openai", "gpt-5.4"))).toBe( + "gpt-5.4", + ); }); test("step headline names the step and how many remain", () => { diff --git a/src/tui/provider/choices.ts b/src/tui/provider/choices.ts index 23397a313..24d83eff9 100644 --- a/src/tui/provider/choices.ts +++ b/src/tui/provider/choices.ts @@ -21,7 +21,7 @@ import { selectableZenModelIds, } from "../../provider/model-catalogs.js"; import { isZenProviderId } from "../../../packages/zen/src/index.js"; -import { buildModelsFirstCatalog } from "../model-catalog.js"; +import { buildModelsFirstCatalog, modelOptionRef } from "../model-catalog.js"; import type { ResidualCatalogEntry } from "../residuals.js"; import type { CliRenderer } from "@opentui/core"; import { createOverlayList } from "../shell/overlay-list.js"; @@ -325,10 +325,15 @@ export function modelChoiceRows( ]; } -/** `provider:model` → `model`, for a row id produced by the model catalog. */ +/** Decode a row id produced by the model catalog for the expected provider. */ export function modelFromRowId(providerId: string, rowId: string): string { - const prefix = `${providerId}:`; - return rowId.startsWith(prefix) ? rowId.slice(prefix.length) : rowId; + if (rowId === TYPE_MODEL_ID) return rowId; + const identity = modelOptionRef(rowId); + if (identity === null || identity.provider !== providerId) + throw new Error( + `Invalid model option identity for provider "${providerId}".`, + ); + return identity.model; } /** diff --git a/src/tui/runner-host.test.ts b/src/tui/runner-host.test.ts index 34a5461c0..db727bb83 100644 --- a/src/tui/runner-host.test.ts +++ b/src/tui/runner-host.test.ts @@ -6,6 +6,7 @@ import type { KeyEvent } from "@opentui/core"; import type { CostSummary } from "../cost/cost-summary.js"; import type { SubAgentSession } from "../subagent/session-store.js"; import { createHarness } from "./harness.js"; +import { modelOptionId, modelOptionRef } from "./model-catalog.js"; import { acceptOverlaySelection, closeInsetOverlay, @@ -15,6 +16,7 @@ import { runOverlayAction, } from "./shell/overlay-list.js"; import { resolvePaletteCatalog } from "./shell/palette.js"; +import { setShellRunState } from "./shell/chrome.js"; import { mountRunnerHost, observeSessionFromSubAgents, @@ -186,6 +188,53 @@ describe("mountRunnerHost session bridge", () => { harness.destroy(); } }); + + test("opens credential recovery only after the shell is idle", async () => { + const harness = await createHarness({ width: 80, height: 24 }); + const host = await mountRunnerHost({ + title: "test", + eventEmitter: new EventEmitter(), + send: () => undefined, + interrupt: () => undefined, + deliver: () => undefined, + providers: {}, + onModelSelect: () => undefined, + commands: [], + onCommand: () => undefined, + chrome: () => ({ agents: [] }), + subscribeChrome: () => () => undefined, + subAgentSessions: () => [], + createRenderer: async () => harness.renderer, + }); + const accepted: string[] = []; + const args = { + alternatives: [ + { + id: modelOptionId("backup", "model-a"), + label: "model-a * [backup]", + provider: "backup", + model: "model-a", + }, + ], + onAccept: (id: string) => accepted.push(id), + onCancel: () => undefined, + }; + try { + host.bridge.beginSystemContinuation("busy"); + expect(host.openCredentialRecovery(args)).toBe(false); + expect(host.shell.overlayKind).toBeNull(); + + setShellRunState(host.shell, "idle"); + expect(host.openCredentialRecovery(args)).toBe(true); + expect(host.shell.overlayKind).toBe("model_picker"); + expect(host.shell.overlayItems).toEqual(["model-a * [backup]"]); + acceptOverlaySelection(host.shell); + expect(accepted).toEqual([modelOptionId("backup", "model-a")]); + } finally { + host.dispose(); + harness.destroy(); + } + }); }); describe("mountRunnerHost chrome wiring", () => { @@ -419,7 +468,7 @@ describe("mountRunnerHost model picker", () => { option: true, } as KeyEvent; expect(runOverlayAction(host.shell, fKey)).toBe(true); - expect(toggled).toEqual(["xai:grok-4"]); + expect(toggled).toEqual([modelOptionId("xai", "grok-4")]); } finally { host.dispose(); harness.destroy(); @@ -454,7 +503,7 @@ describe("mountRunnerHost model picker", () => { option: true, } as KeyEvent; expect(runOverlayAction(host.shell, dKey)).toBe(true); - expect(setDefault).toEqual(["xai:grok-4"]); + expect(setDefault).toEqual([modelOptionId("xai", "grok-4")]); } finally { host.dispose(); harness.destroy(); @@ -644,9 +693,8 @@ describe("bottom border cost run", () => { "codex/abk-labs": { models: ["gpt-5.5"] }, }, onModelSelect: (id) => { - const sep = id.indexOf(":"); - if (sep <= 0) return; - provider = id.slice(0, sep); + const identity = modelOptionRef(id); + if (identity !== null) provider = identity.provider; }, commands: [], onCommand: () => undefined, @@ -698,9 +746,8 @@ describe("bottom border cost run", () => { xai: { models: ["grok-4"] }, }, onModelSelect: (id) => { - const sep = id.indexOf(":"); - if (sep <= 0) return; - provider = id.slice(0, sep); + const identity = modelOptionRef(id); + if (identity !== null) provider = identity.provider; }, commands: [], onCommand: () => undefined, diff --git a/src/tui/runner/credential-recovery.test.ts b/src/tui/runner/credential-recovery.test.ts new file mode 100644 index 000000000..dc9cee73a --- /dev/null +++ b/src/tui/runner/credential-recovery.test.ts @@ -0,0 +1,560 @@ +import { describe, expect, test } from "bun:test"; +import type { InboundMessage } from "@intx/types/runtime"; +import { + applyCredentialRecoverySelection, + buildCredentialRecoveryAlternatives, + buildCredentialRecoveryContinuationMessage, + createCredentialRecoveryState, +} from "./credential-recovery.js"; +import { modelOptionId, modelOptionRef } from "../model-catalog.js"; + +function operatorMessage(): InboundMessage { + return { + ref: { uid: 1, mailbox: "INBOX" }, + headers: { + from: "user@local", + to: ["agent@local"], + date: "2026-09-26T00:00:00.000Z", + messageId: "", + interchangeType: "conversation.message", + }, + flags: ["operator-originated"], + signatureStatus: "missing", + content: "inspect this", + attachments: [ + { + name: "screen.png", + contentType: "image/png", + data: new Uint8Array([1, 2, 3]), + }, + ], + }; +} + +function required(value: T | null, label: string): T { + if (value === null) throw new Error(`expected ${label}`); + return value; +} + +const providers = [ + { + name: "failed", + baseURL: "https://failed.test/v1", + apiKey: "failed-key", + models: ["same", "other"], + }, + { + name: "backup", + baseURL: "https://backup.test/v1", + apiKey: "backup-key", + models: ["model-a", "model-a", "model-b"], + }, + { + name: "broken", + baseURL: "", + apiKey: "", + models: ["not-assemblable"], + }, +]; + +describe("credential recovery alternatives", () => { + test("exclude the failed provider, deduplicate, and retain only assemblable rows", () => { + expect( + buildCredentialRecoveryAlternatives( + { + providers, + providerName: "failed", + model: "same", + } as never, + "session-1", + "failed", + ).map(({ id, provider, model }) => ({ id, provider, model })), + ).toEqual([ + { + id: modelOptionId("backup", "model-a"), + provider: "backup", + model: "model-a", + }, + { + id: modelOptionId("backup", "model-b"), + provider: "backup", + model: "model-b", + }, + ]); + }); + + test("keeps colon-bearing provider and model identities distinct", () => { + const alternatives = buildCredentialRecoveryAlternatives( + { + providers: [ + providers[0], + { + name: "backup:west", + baseURL: "https://west.test/v1", + apiKey: "west-key", + models: ["model:fast"], + }, + { + name: "backup", + baseURL: "https://backup.test/v1", + apiKey: "backup-key", + models: ["west:model:fast"], + }, + ], + providerName: "failed", + model: "same", + } as never, + "session-colons", + "failed", + ); + + expect(alternatives).toHaveLength(2); + expect(alternatives.map((alternative) => alternative.id)).toEqual([ + modelOptionId("backup:west", "model:fast"), + modelOptionId("backup", "west:model:fast"), + ]); + }); +}); + +describe("generation-scoped credential recovery", () => { + test("arms only after a repeated terminal credential failure and preserves the original input", () => { + const state = createCredentialRecoveryState(); + const message = operatorMessage(); + const attempt = state.begin(message, "failed"); + + state.observe(attempt, { + type: "inference.retry", + data: { + previousError: { category: "credential_failure", message: "401" }, + }, + }); + state.observe(attempt, { + type: "inference.error", + data: { error: { category: "credential_failure", message: "still 401" } }, + }); + + const pending = state.settle(attempt, [ + { + id: modelOptionId("backup", "model-a"), + label: "model-a * [backup]", + provider: "backup", + model: "model-a", + }, + ]); + expect(pending?.message).toBe(message); + expect(pending?.message.attachments).toEqual(message.attachments); + }); + + test("does not arm for one credential failure, a noncredential terminal, or no alternatives", () => { + const state = createCredentialRecoveryState(); + const oneFailure = state.begin(operatorMessage(), "failed"); + state.observe(oneFailure, { + type: "inference.error", + data: { error: { category: "credential_failure", message: "401" } }, + }); + expect(state.settle(oneFailure, [])).toBeNull(); + + const otherFailure = state.begin(operatorMessage(), "failed"); + state.observe(otherFailure, { + type: "inference.retry", + data: { + previousError: { category: "credential_failure", message: "401" }, + }, + }); + state.observe(otherFailure, { + type: "inference.error", + data: { error: { category: "fatal", message: "boom" } }, + }); + expect( + state.settle(otherFailure, [ + { + id: modelOptionId("backup", "model-a"), + label: "backup", + provider: "backup", + model: "model-a", + }, + ]), + ).toBeNull(); + }); + + test("cancel consumes the matching generation without switching or replaying", () => { + const state = createCredentialRecoveryState(); + const attempt = state.begin(operatorMessage(), "failed"); + state.observe(attempt, { + type: "inference.retry", + data: { + previousError: { category: "credential_failure", message: "401" }, + }, + }); + state.observe(attempt, { + type: "inference.error", + data: { error: { category: "credential_failure", message: "401" } }, + }); + const pending = required( + state.settle(attempt, [ + { + id: modelOptionId("backup", "model-a"), + label: "backup", + provider: "backup", + model: "model-a", + }, + ]), + "pending recovery", + ); + expect(state.cancel(pending.generation)).toBe(true); + expect( + state.accept(pending.generation, modelOptionId("backup", "model-a")), + ).toEqual({ + kind: "stale", + }); + }); + + test("accept consumes once, validates generation, and vetoes replay after commitment", () => { + const state = createCredentialRecoveryState(); + const first = state.begin(operatorMessage(), "failed"); + state.observe(first, { + type: "inference.retry", + data: { + previousError: { category: "credential_failure", message: "401" }, + }, + }); + state.observe(first, { + type: "inference.error", + data: { error: { category: "credential_failure", message: "401" } }, + }); + const pending = required( + state.settle(first, [ + { + id: modelOptionId("backup", "model-a"), + label: "backup", + provider: "backup", + model: "model-a", + }, + ]), + "pending recovery", + ); + + expect( + state.accept(pending.generation + 1, modelOptionId("backup", "model-a")), + ).toEqual({ + kind: "stale", + }); + expect( + state.accept(pending.generation, modelOptionId("backup", "model-a")), + ).toEqual({ + kind: "accepted", + alternative: required(pending.alternatives[0] ?? null, "alternative"), + replay: true, + generation: pending.generation, + }); + expect( + state.accept(pending.generation, modelOptionId("backup", "model-a")), + ).toEqual({ + kind: "stale", + }); + + const committed = state.begin(operatorMessage(), "failed"); + state.observe(committed, { + type: "inference.retry", + data: { + previousError: { category: "credential_failure", message: "401" }, + }, + }); + state.observe(committed, { + type: "inference.text.delta", + data: { delta: "x" }, + }); + state.observe(committed, { + type: "inference.error", + data: { error: { category: "credential_failure", message: "401" } }, + }); + const committedPending = required( + state.settle(committed, [ + { + id: modelOptionId("backup", "model-a"), + label: "backup", + provider: "backup", + model: "model-a", + }, + ]), + "committed pending recovery", + ); + expect( + state.accept( + committedPending.generation, + modelOptionId("backup", "model-a"), + ), + ).toMatchObject({ kind: "accepted", replay: false }); + }); + + test("a new generation invalidates an older selection and invalid rows consume without replay", () => { + const state = createCredentialRecoveryState(); + const old = state.begin(operatorMessage(), "failed"); + state.observe(old, { + type: "inference.retry", + data: { + previousError: { category: "credential_failure", message: "401" }, + }, + }); + state.observe(old, { + type: "inference.error", + data: { error: { category: "credential_failure", message: "401" } }, + }); + const oldPending = required( + state.settle(old, [ + { + id: modelOptionId("backup", "model-a"), + label: "backup", + provider: "backup", + model: "model-a", + }, + ]), + "old pending recovery", + ); + state.begin(operatorMessage(), "failed"); + expect( + state.accept(oldPending.generation, modelOptionId("backup", "model-a")), + ).toEqual({ + kind: "stale", + }); + + const current = state.begin(operatorMessage(), "failed"); + state.observe(current, { + type: "inference.retry", + data: { + previousError: { category: "credential_failure", message: "401" }, + }, + }); + state.observe(current, { + type: "inference.error", + data: { error: { category: "credential_failure", message: "401" } }, + }); + const pending = required( + state.settle(current, [ + { + id: modelOptionId("backup", "model-a"), + label: "backup", + provider: "backup", + model: "model-a", + }, + ]), + "current pending recovery", + ); + expect( + state.accept(pending.generation, modelOptionId("missing", "model")), + ).toEqual({ + kind: "invalid", + }); + expect( + state.accept(pending.generation, modelOptionId("backup", "model-a")), + ).toEqual({ + kind: "stale", + }); + }); + + test("a background continuation preserves an actionable operator selection", () => { + const { state, pending } = pendingRecovery(); + + state.begin(buildCredentialRecoveryContinuationMessage(99), "failed"); + + expect( + state.accept(pending.generation, modelOptionId("backup", "model-a")), + ).toMatchObject({ + kind: "accepted", + replay: true, + }); + }); +}); + +function pendingRecovery(committed = false) { + const state = createCredentialRecoveryState(); + const attempt = state.begin(operatorMessage(), "failed"); + state.observe(attempt, { + type: "inference.retry", + data: { previousError: { category: "credential_failure" } }, + }); + if (committed) { + state.observe(attempt, { + type: "inference.thinking.delta", + data: { delta: "thinking" }, + }); + } + state.observe(attempt, { + type: "inference.error", + data: { error: { category: "credential_failure" } }, + }); + const pending = state.settle(attempt, [ + { + id: modelOptionId("backup", "model-a"), + label: "backup", + provider: "backup", + model: "model-a", + }, + ]); + if (pending === null) throw new Error("expected pending recovery"); + return { state, pending }; +} + +describe("credential recovery selection effects", () => { + test("applies the exact colon-bearing pair and replays once", () => { + const state = createCredentialRecoveryState(); + const attempt = state.begin(operatorMessage(), "failed"); + state.observe(attempt, { + type: "inference.retry", + data: { previousError: { category: "credential_failure" } }, + }); + state.observe(attempt, { + type: "inference.error", + data: { error: { category: "credential_failure" } }, + }); + const id = modelOptionId("backup:west", "model:fast"); + const alternative = { + id, + label: "model:fast * [backup:west]", + provider: "backup:west", + model: "model:fast", + }; + const pending = required(state.settle(attempt, [alternative]), "pending"); + const switched: (typeof alternative)[] = []; + const delivered: InboundMessage[] = []; + + expect( + applyCredentialRecoverySelection({ + state, + generation: pending.generation, + alternativeId: id, + switchAlternative: (selected) => switched.push(selected), + armContinuation: () => undefined, + cancelContinuation: () => undefined, + deliverContinuation: (message) => delivered.push(message), + }), + ).toBe("continued"); + expect(switched).toEqual([alternative]); + expect(modelOptionRef(switched[0]?.id ?? "")).toEqual({ + provider: "backup:west", + model: "model:fast", + }); + expect(delivered).toHaveLength(1); + expect(state.accept(pending.generation, id)).toEqual({ kind: "stale" }); + }); + + test("switches and continues an uncommitted input exactly once", () => { + const { state, pending } = pendingRecovery(); + const switches: string[] = []; + const arms: number[] = []; + const deliveries: InboundMessage[] = []; + const args = { + state, + generation: pending.generation, + alternativeId: modelOptionId("backup", "model-a"), + switchAlternative: (alternative: { id: string }) => + switches.push(alternative.id), + armContinuation: (generation: number) => arms.push(generation), + cancelContinuation: () => undefined, + deliverContinuation: (message: InboundMessage) => + deliveries.push(message), + }; + + expect(applyCredentialRecoverySelection(args)).toBe("continued"); + expect(applyCredentialRecoverySelection(args)).toBe("stale"); + expect(switches).toEqual([modelOptionId("backup", "model-a")]); + expect(arms).toEqual([pending.generation]); + expect(deliveries).toHaveLength(1); + expect(deliveries[0]?.content).toBe(""); + }); + + test("committed input switches future source without continuation", () => { + const { state, pending } = pendingRecovery(true); + let switches = 0; + let deliveries = 0; + expect( + applyCredentialRecoverySelection({ + state, + generation: pending.generation, + alternativeId: modelOptionId("backup", "model-a"), + switchAlternative: () => switches++, + armContinuation: () => { + throw new Error("must not arm"); + }, + cancelContinuation: () => undefined, + deliverContinuation: () => deliveries++, + }), + ).toBe("switched"); + expect(switches).toBe(1); + expect(deliveries).toBe(0); + }); + + test("switch and continuation failures consume without cascading", () => { + const failedSwitch = pendingRecovery(); + let arms = 0; + expect( + applyCredentialRecoverySelection({ + state: failedSwitch.state, + generation: failedSwitch.pending.generation, + alternativeId: modelOptionId("backup", "model-a"), + switchAlternative: () => { + throw new Error("unavailable"); + }, + armContinuation: () => arms++, + cancelContinuation: () => undefined, + deliverContinuation: () => { + throw new Error("must not deliver"); + }, + }), + ).toBe("switch-failed"); + expect(arms).toBe(0); + + const failedDelivery = pendingRecovery(); + const cancelled: number[] = []; + expect( + applyCredentialRecoverySelection({ + state: failedDelivery.state, + generation: failedDelivery.pending.generation, + alternativeId: modelOptionId("backup", "model-a"), + switchAlternative: () => undefined, + armContinuation: () => undefined, + cancelContinuation: (generation) => cancelled.push(generation), + deliverContinuation: () => { + throw new Error("closed"); + }, + }), + ).toBe("switched"); + expect(cancelled).toEqual([failedDelivery.pending.generation]); + expect( + failedDelivery.state.accept( + failedDelivery.pending.generation, + modelOptionId("backup", "model-a"), + ), + ).toEqual({ kind: "stale" }); + }); + + test("an invalid identity cannot switch or arm continuation", () => { + const { state, pending } = pendingRecovery(); + let switches = 0; + let arms = 0; + expect( + applyCredentialRecoverySelection({ + state, + generation: pending.generation, + alternativeId: "not-an-option-id", + switchAlternative: () => switches++, + armContinuation: () => arms++, + cancelContinuation: () => undefined, + deliverContinuation: () => { + throw new Error("must not deliver"); + }, + }), + ).toBe("invalid"); + expect(switches).toBe(0); + expect(arms).toBe(0); + }); +}); + +test("credential recovery continuation is a dedicated contentless system inbound", () => { + const message = buildCredentialRecoveryContinuationMessage(7); + expect(message.ref).toEqual({ uid: 0, mailbox: "system" }); + expect(message.content).toBe(""); + expect(message.attachments).toBeUndefined(); + expect(message.headers.interchangeType).toBe("system.credential.refresh"); + expect(message.headers.interchangeCorrelationId).toBe("7"); +}); diff --git a/src/tui/runner/credential-recovery.ts b/src/tui/runner/credential-recovery.ts new file mode 100644 index 000000000..d8f4bd024 --- /dev/null +++ b/src/tui/runner/credential-recovery.ts @@ -0,0 +1,268 @@ +import type { InboundMessage } from "@intx/types/runtime"; +import type { Config } from "../../config/index.js"; +import { buildMainSessionSources } from "../../config/inference-sources.js"; +import { isOperatorOriginated } from "../../agent/message-provenance.js"; +import { CREDENTIAL_RECOVERY_INTERCHANGE_TYPE } from "../../agent/director.js"; +import { + formatModelPickerLabel, + modelOptionId, + modelOptionRef, +} from "../model-catalog.js"; + +export interface CredentialRecoveryAlternative { + readonly id: string; + readonly label: string; + readonly provider: string; + readonly model: string; +} + +export interface PendingCredentialRecovery { + readonly generation: number; + readonly failedProvider: string; + readonly message: InboundMessage; + readonly committed: boolean; + readonly alternatives: readonly CredentialRecoveryAlternative[]; +} + +export interface CredentialRecoveryAttempt { + readonly generation: number; + failedProvider: string; + readonly message: InboundMessage; + committed: boolean; + sawCredentialRetry: boolean; + terminalCredentialFailure: boolean; +} + +type RecoveryEvent = { + readonly type: string; + readonly data?: unknown; +}; + +type RecoveryEventData = { + readonly previousError?: { readonly category?: string }; + readonly error?: { + readonly category?: string; + readonly providerId?: string; + }; +}; + +function recoveryEventData( + event: RecoveryEvent, +): RecoveryEventData | undefined { + if (typeof event.data !== "object" || event.data === null) return undefined; + return event.data as RecoveryEventData; +} + +export type CredentialRecoveryAcceptance = + | { readonly kind: "stale" } + | { readonly kind: "invalid" } + | { + readonly kind: "accepted"; + readonly generation: number; + readonly alternative: CredentialRecoveryAlternative; + readonly replay: boolean; + }; + +function isHarnessCommittingEvent(event: RecoveryEvent): boolean { + if (!event.type.startsWith("inference.")) return false; + // Keep the exact harness.ts isCommitting set. runInference handles these + // wrapper terminal events before consulting that predicate. + switch (event.type) { + case "inference.start": + case "inference.usage": + case "inference.done": + case "inference.error": + case "inference.retry": + return false; + default: + return true; + } +} + +export function createCredentialRecoveryState() { + let nextGeneration = 0; + let pending: PendingCredentialRecovery | null = null; + + return { + begin( + message: InboundMessage, + failedProvider: string, + ): CredentialRecoveryAttempt { + if (isOperatorOriginated(message.flags)) pending = null; + return { + generation: ++nextGeneration, + failedProvider, + message, + committed: false, + sawCredentialRetry: false, + terminalCredentialFailure: false, + }; + }, + observe(attempt: CredentialRecoveryAttempt, event: RecoveryEvent): void { + if (attempt.generation !== nextGeneration) return; + if (isHarnessCommittingEvent(event)) attempt.committed = true; + const data = recoveryEventData(event); + if ( + event.type === "inference.retry" && + data?.previousError?.category === "credential_failure" + ) { + attempt.sawCredentialRetry = true; + } + if (event.type === "inference.error") { + attempt.terminalCredentialFailure = + data?.error?.category === "credential_failure"; + const providerId = data?.error?.providerId; + if (providerId !== undefined && providerId.length > 0) { + attempt.failedProvider = providerId; + } + } + }, + settle( + attempt: CredentialRecoveryAttempt, + alternatives: readonly CredentialRecoveryAlternative[], + ): PendingCredentialRecovery | null { + if ( + attempt.generation !== nextGeneration || + !isOperatorOriginated(attempt.message.flags) || + !attempt.sawCredentialRetry || + !attempt.terminalCredentialFailure || + alternatives.length === 0 + ) { + return null; + } + pending = { + generation: attempt.generation, + failedProvider: attempt.failedProvider, + message: attempt.message, + committed: attempt.committed, + alternatives: [...alternatives], + }; + return pending; + }, + cancel(generation: number): boolean { + if (pending?.generation !== generation) return false; + pending = null; + return true; + }, + accept( + generation: number, + alternativeId: string, + ): CredentialRecoveryAcceptance { + if (pending?.generation !== generation) return { kind: "stale" }; + const claimed = pending; + pending = null; + const identity = modelOptionRef(alternativeId); + if (identity === null) return { kind: "invalid" }; + const alternative = claimed.alternatives.find( + (candidate) => + candidate.id === alternativeId && + candidate.provider === identity.provider && + candidate.model === identity.model, + ); + if (alternative === undefined) return { kind: "invalid" }; + return { + kind: "accepted", + generation, + alternative, + replay: !claimed.committed, + }; + }, + clear(): void { + pending = null; + nextGeneration++; + }, + }; +} + +export function applyCredentialRecoverySelection(args: { + state: ReturnType; + generation: number; + alternativeId: string; + switchAlternative: (alternative: CredentialRecoveryAlternative) => void; + armContinuation: (generation: number) => void; + cancelContinuation: (generation: number) => void; + deliverContinuation: (message: InboundMessage) => void; +}): "stale" | "invalid" | "switch-failed" | "switched" | "continued" { + const acceptance = args.state.accept(args.generation, args.alternativeId); + if (acceptance.kind !== "accepted") return acceptance.kind; + try { + args.switchAlternative(acceptance.alternative); + } catch { + return "switch-failed"; + } + if (!acceptance.replay) return "switched"; + args.armContinuation(acceptance.generation); + try { + args.deliverContinuation( + buildCredentialRecoveryContinuationMessage(acceptance.generation), + ); + } catch { + args.cancelContinuation(acceptance.generation); + return "switched"; + } + return "continued"; +} + +export function buildCredentialRecoveryAlternatives( + config: Pick< + Config, + "providers" | "providerName" | "model" | "settings" | "reasoningEffort" + >, + sessionId: string, + failedProvider: string, +): CredentialRecoveryAlternative[] { + const alternatives: CredentialRecoveryAlternative[] = []; + const seen = new Set(); + + for (const entry of config.providers) { + if (entry.name === failedProvider) continue; + for (const rawModel of entry.models ?? []) { + const model = rawModel.trim(); + if (model.length === 0) continue; + const id = modelOptionId(entry.name, model); + if (seen.has(id)) continue; + try { + buildMainSessionSources({ + settings: config.settings, + catalog: config.providers, + activeProvider: entry.name, + activeModel: model, + sessionId, + ...(config.reasoningEffort !== undefined + ? { reasoningEffort: config.reasoningEffort } + : {}), + }); + } catch { + continue; + } + seen.add(id); + alternatives.push({ + id, + provider: entry.name, + model, + label: formatModelPickerLabel(model, entry.name), + }); + } + } + + return alternatives; +} + +export function buildCredentialRecoveryContinuationMessage( + generation: number, +): InboundMessage { + return { + ref: { uid: 0, mailbox: "system" }, + headers: { + from: "user@local", + to: ["agent@local"], + date: new Date().toISOString(), + messageId: `credential-recovery-${generation}@local`, + interchangeType: CREDENTIAL_RECOVERY_INTERCHANGE_TYPE, + interchangeCorrelationId: String(generation), + }, + flags: [], + content: "", + signatureStatus: "missing", + }; +} diff --git a/src/tui/runner/exit.test.ts b/src/tui/runner/exit.test.ts index 19a0997ae..171829c02 100644 --- a/src/tui/runner/exit.test.ts +++ b/src/tui/runner/exit.test.ts @@ -5,6 +5,11 @@ import { getLogger } from "@intx/log"; import type { InferenceSource } from "@intx/types/runtime"; import * as codexSession from "../../auth/codex/session.js"; +import * as xaiSession from "../../auth/xai/session.js"; +import { + readSourceCredentialMaterial, + registerSourceCredentialRecord, +} from "../../config/source-credentials.js"; import { createChatDirector } from "../../agent/director.js"; import * as sessionIndex from "../../session/index.js"; import { createSubAgentSessionStore } from "../../subagent/session-store.js"; @@ -334,6 +339,10 @@ function stubSendLifecycle(agent: Agent): { state: RunnerState; services: RunnerServices; } { + registerSourceCredentialRecord(liveSource.credentialId, { + provenance: { kind: "oauth", provider: "codex", profile: "work" }, + material: { secret: "stale-token" }, + }); const state = { runTaskTitle: "keep-title", liveSource, @@ -344,8 +353,6 @@ function stubSendLifecycle(agent: Agent): { inFlight: 0, fatalBuildError: null, sendAborted: false, - initialCodexProfile: "work", - initialXaiProfile: undefined, stampProvider: { fn: undefined }, } as unknown as RunnerState; const services = { @@ -432,6 +439,61 @@ describe("agentProxy.send vs /clear", () => { hung.spy.mockRestore(); } }); + + for (const provider of ["codex", "xai"] as const) { + test(`${provider}/shadow API-key send never resolves same-slug OAuth`, async () => { + const sends: string[] = []; + const { state, services } = stubSendLifecycle(recordingAgent(sends)); + const source: InferenceSource = { + id: `${provider}/shadow`, + provider: "openai-compatible", + baseURL: "https://relay.example/v1", + credentialId: `${provider}/shadow`, + model: "relay-model", + }; + state.liveSource = source; + state.config = { + ...state.config, + providers: [ + { + name: `${provider}/shadow`, + baseURL: "https://oauth.example/v1", + apiKey: "oauth-token", + models: ["relay-model"], + ...(provider === "codex" + ? { codexProfile: "shadow" } + : { xaiProfile: "shadow" }), + }, + ], + }; + registerSourceCredentialRecord(source.credentialId, { + provenance: { kind: "api-key" }, + material: { secret: "explicit-api-key" }, + }); + const codexResolver = spyOn( + codexSession, + "getValidCodexToken", + ).mockRejectedValue(new Error("must not resolve Codex OAuth")); + const xaiResolver = spyOn( + xaiSession, + "getValidXaiToken", + ).mockRejectedValue(new Error("must not resolve xAI OAuth")); + + try { + const { agentProxy } = await createRunLifecycle(state, services); + await agentProxy.send("use explicit authorization"); + expect(sends).toEqual(["use explicit authorization"]); + expect(codexResolver).not.toHaveBeenCalled(); + expect(xaiResolver).not.toHaveBeenCalled(); + expect(readSourceCredentialMaterial(source.credentialId).secret).toBe( + "explicit-api-key", + ); + } finally { + codexResolver.mockRestore(); + xaiResolver.mockRestore(); + } + }); + } }); const rebuildMockState: ReactorState = {} as unknown as ReactorState; diff --git a/src/tui/runner/exit.ts b/src/tui/runner/exit.ts index f5d07c840..f6bc400f2 100644 --- a/src/tui/runner/exit.ts +++ b/src/tui/runner/exit.ts @@ -11,7 +11,6 @@ import { type Agent, } from "@intx/agent"; import { getLogger } from "@intx/log"; -import type { InferenceSource } from "@intx/types/runtime"; import { consumeStream } from "../../session/stream-consumer.js"; import { COMPACTION_CONTINUATION_EVENT } from "../../agent/compaction.js"; import { @@ -42,16 +41,11 @@ import { printResumeHint } from "../../session/resume-hint.js"; import { clearActiveDisposeHost } from "../../session/active-host.js"; import { syncRunStateHandle } from "../../session/active-run.js"; import { startRunHeartbeat } from "../../session/run-liveness.js"; -import { getValidCodexToken } from "../../auth/codex/session.js"; -import { getValidXaiToken } from "../../auth/xai/session.js"; import { suppressProviderFailurePresentation } from "../provider/failure-attempt.js"; import { normalizeInferenceErrorForTerminal } from "../../inference-gateway-error.js"; -import { codexProfileFromProviderName } from "../../config/codex-providers.js"; -import { xaiProfileFromProviderName } from "../../config/xai-providers.js"; -import { - peekSourceCredentialSecret, - registerSourceCredential, -} from "../../config/source-credentials.js"; +import { ensureFreshInferenceSource } from "../../subagent/refresh-inference-source.js"; +import { peekSourceCredentialSecret } from "../../config/source-credentials.js"; +import { sanitizeDiagnosticValue } from "../../diagnostic-sanitize.js"; import { LOG_NAMESPACE_ROOT } from "../../branding.js"; import { cancelFeedbackCapture } from "../../telemetry/feedback.js"; import { @@ -316,6 +310,14 @@ export async function createRunLifecycle( providerFailureAttempts.advanceToNextMessage(); } services.correlationAcceptance.observe(event); + const providerAttempt = providerFailureAttempts.current(); + const recoveryAttempt = + providerAttempt === undefined + ? undefined + : state.credentialRecoveryAttempts.get(providerAttempt); + if (recoveryAttempt !== undefined) { + state.credentialRecovery.observe(recoveryAttempt, event); + } if (event.type === "inference.start" || event.type === "inference.done") { providerFailureAttempts.reset(); } else if (event.type === "inference.error") { @@ -361,8 +363,17 @@ export async function createRunLifecycle( }, (message, fields) => tuiLogger.debug(message, fields), ); - services.runSink.sink(eventForSink); - services.cycleRecorder.handleEvent(event); + const configuredSecret = peekSourceCredentialSecret( + state.liveSource.credentialId, + ); + const sanitizedEventForSink = sanitizeDiagnosticValue(eventForSink, [ + configuredSecret, + ]) as typeof eventForSink; + const sanitizedEvent = sanitizeDiagnosticValue(event, [ + configuredSecret, + ]) as typeof event; + services.runSink.sink(sanitizedEventForSink); + services.cycleRecorder.handleEvent(sanitizedEvent); if (onTurnBoundary(event)) { services.sessionCost.addTurn( event.data.usage, @@ -438,51 +449,11 @@ export async function createRunLifecycle( }; services.toolset.setToolPromoter(promoteTools); - // The active Codex source, tracked whenever a "codex/" source is - // selected so its access token can be refreshed before each send. Seeded from - // config when the session starts on a Codex profile (buildAgent sets that - // source directly, not through the proxy's setSource). - state.activeCodexSource = - state.initialCodexProfile !== undefined - ? { profile: state.initialCodexProfile, source: state.liveSource } - : undefined; - state.activeXaiSource = - state.initialXaiProfile !== undefined - ? { profile: state.initialXaiProfile, source: state.liveSource } - : undefined; - - // Refresh the active Codex access token (if any) and push it onto the live - // agent before a send. getValidCodexToken returns the stored token when still - // valid and refreshes transparently otherwise, so this satisfies "check - // before each inference call" without crashing the loop: a failure surfaces - // as a CodexAuthError naming the profile and rejects the send. - // - // The source is pushed on every send, not only when the token changed: an - // agent rebuild (interrupt, /clear) reseeds the source from - // the original login-time token, so unconditionally re-pushing the live token - // is what keeps the rebuilt agent from sending a stale credential. - const refreshCodexBeforeSend = async (): Promise => { - const active = state.activeCodexSource; - if (active === undefined) return; - const { access } = await getValidCodexToken(active.profile); - const source: InferenceSource = active.source; - if (access !== peekSourceCredentialSecret(source.credentialId)) { - registerSourceCredential(source.credentialId, access); - } - state.activeCodexSource = { profile: active.profile, source }; - state.liveSource = source; - setAgentSourceUnlessClosed(liveAgent(state), source); - }; - - const refreshXaiBeforeSend = async (): Promise => { - const active = state.activeXaiSource; - if (active === undefined) return; - const { access } = await getValidXaiToken(active.profile); - const source: InferenceSource = active.source; - if (access !== peekSourceCredentialSecret(source.credentialId)) { - registerSourceCredential(source.credentialId, access); - } - state.activeXaiSource = { profile: active.profile, source }; + const refreshBeforeSend = async (): Promise => { + const source = await ensureFreshInferenceSource( + state.liveSource, + state.config.providers, + ); state.liveSource = source; setAgentSourceUnlessClosed(liveAgent(state), source); }; @@ -512,8 +483,7 @@ export async function createRunLifecycle( void persistRunSnapshot("running"); } return await runWhileAgentBusy(state, async () => { - await refreshCodexBeforeSend(); - await refreshXaiBeforeSend(); + await refreshBeforeSend(); dropIfRotated(); return await liveAgent(state).send(content, opts); }); @@ -525,14 +495,6 @@ export async function createRunLifecycle( }, close: () => liveAgent(state).close(), setSource: (source) => { - const codexProfile = codexProfileFromProviderName(source.id); - const xaiProfile = xaiProfileFromProviderName(source.id); - state.activeCodexSource = - codexProfile !== undefined - ? { profile: codexProfile, source } - : undefined; - state.activeXaiSource = - xaiProfile !== undefined ? { profile: xaiProfile, source } : undefined; state.liveSource = source; state.liveSources = [source]; state.liveDefaultSource = source.id; @@ -546,16 +508,6 @@ export async function createRunLifecycle( state.liveDefaultSource = defaultSource; const head = sources.find((s) => s.id === defaultSource) ?? sources[0]; if (head !== undefined) { - const codexProfile = codexProfileFromProviderName(head.id); - const xaiProfile = xaiProfileFromProviderName(head.id); - state.activeCodexSource = - codexProfile !== undefined - ? { profile: codexProfile, source: head } - : undefined; - state.activeXaiSource = - xaiProfile !== undefined - ? { profile: xaiProfile, source: head } - : undefined; state.liveSource = head; state.stampProvider.fn?.(head.id); } @@ -580,6 +532,7 @@ export async function createRunLifecycle( // Close it, drain the old stream, and rebuild a fresh agent so the next send // works. const interrupt = (): void => { + state.credentialRecovery?.clear(); // CL-8220 gate: an in-flight compact runs inline on the vendored reactor // with no abort hop of its own, so an interrupt that merely queues behind // it parks until the summary call returns. Abort the compact first — the @@ -651,6 +604,7 @@ export async function createRunLifecycle( // abort handles → child agent.close) before clearing the session store so // /clear does not leave orphaned child reactors burning tokens. const newSession = (): void => { + state.credentialRecovery?.clear(); // CL-8220: rotation must not park behind an in-flight compact either. state.compactionLifecycle?.abortCompaction("session rotation"); const cancelledWorkers = resetSessionForRotation(state, services); diff --git a/src/tui/runner/host.ts b/src/tui/runner/host.ts index a637cbdb8..95a6c6567 100644 --- a/src/tui/runner/host.ts +++ b/src/tui/runner/host.ts @@ -60,6 +60,8 @@ import { pushToolCall, pushToolResult } from "../tool-rows.js"; import type { StreamRow } from "../stream.js"; import type { QueueKind } from "../delivery-queue.js"; import type { ShellOutputFeed } from "../../session/shell-output-feed.js"; +import { openModelPickerOverlay } from "../overlays.js"; +import type { CredentialRecoveryAlternative } from "./credential-recovery.js"; export interface RunnerHostDeps { readonly title: string; @@ -182,6 +184,11 @@ export type RunnerHost = ProductHost & { ) => void; /** Re-reads `showPromptCost` and cost/context state, repainting the border immediately. */ readonly refreshCostContext: () => void; + readonly openCredentialRecovery: (args: { + alternatives: readonly CredentialRecoveryAlternative[]; + onAccept: (id: string) => void; + onCancel: () => void; + }) => boolean; }; /** Map a subagent transcript entry to a stream row. */ @@ -432,6 +439,24 @@ export async function mountRunnerHost( notify: (text) => surfaceSystemNotice(host.shell, text), }; + const openCredentialRecovery: RunnerHost["openCredentialRecovery"] = ( + args, + ) => { + if (host.shell.session.run !== "idle" || args.alternatives.length === 0) { + return false; + } + openModelPickerOverlay(host.shell, { + items: args.alternatives.map((alternative) => alternative.label), + itemIds: args.alternatives.map((alternative) => alternative.id), + typeToFilter: true, + onCancel: args.onCancel, + onAccept: (selection) => { + if (selection.id !== undefined) args.onAccept(selection.id); + }, + }); + return true; + }; + const refreshModels = ( recentModels: readonly ModelCatalogRef[], favoriteModels: readonly ModelCatalogRef[], @@ -452,5 +477,6 @@ export async function mountRunnerHost( openSurface: (kind) => openCommandSurface(host.shell, kind, surfaceDeps), refreshModels, refreshCostContext: pushCostContext, + openCredentialRecovery, }; } diff --git a/src/tui/runner/index.ts b/src/tui/runner/index.ts index 4594f40a7..07eb3e0f6 100644 --- a/src/tui/runner/index.ts +++ b/src/tui/runner/index.ts @@ -35,7 +35,8 @@ import { } from "./submit.js"; import { wireMcp } from "./mcp.js"; import { wirePostStartup } from "./wiring.js"; -import { createRunnerState } from "./state.js"; +import { createRunnerState, liveAgent } from "./state.js"; +import { applyCredentialRecoverySelection } from "./credential-recovery.js"; import { getLogger } from "@intx/log"; import { LOG_NAMESPACE_ROOT } from "../../branding.js"; @@ -220,6 +221,34 @@ export async function runTUI(initialConfig: Config): Promise { }); state.host = host; services.hostHolder.instance = host; + state.presentCredentialRecovery = (pending) => { + const opened = host.openCredentialRecovery({ + alternatives: pending.alternatives, + onCancel: () => { + state.credentialRecovery.cancel(pending.generation); + }, + onAccept: (id) => { + const director = services.directorHolder.instance; + if (director === undefined) { + state.credentialRecovery.cancel(pending.generation); + return; + } + applyCredentialRecoverySelection({ + state: state.credentialRecovery, + generation: pending.generation, + alternativeId: id, + switchAlternative: (alternative) => + settings.onModelSelect(alternative.id), + armContinuation: (generation) => + director.armCredentialRecoveryContinuation(generation), + cancelContinuation: (generation) => + director.cancelCredentialRecoveryContinuation(generation), + deliverContinuation: (message) => liveAgent(state).deliver(message), + }); + }, + }); + if (!opened) state.credentialRecovery.cancel(pending.generation); + }; wirePostStartup(state, services, mcp.mcpConnectCallbacks); diff --git a/src/tui/runner/settings.ts b/src/tui/runner/settings.ts index f9d72b855..c3be45e12 100644 --- a/src/tui/runner/settings.ts +++ b/src/tui/runner/settings.ts @@ -37,7 +37,7 @@ import pkg from "../../../package.json" with { type: "json" }; import type { GrantScope } from "../../permission/types.js"; import { connectProviderInline } from "../provider/connect.js"; import { persistConnectedSelection } from "../provider/submit.js"; -import { modelOptionId } from "../model-catalog.js"; +import { modelOptionId, modelOptionRef } from "../model-catalog.js"; import { prefetchGoModels, prefetchZenModels, @@ -385,10 +385,9 @@ export async function wireSettings( }; const onModelSelect = (id: string): void => { - const sep = id.indexOf(":"); - if (sep <= 0) return; - const provider = id.slice(0, sep); - const model = id.slice(sep + 1); + const identity = modelOptionRef(id); + if (identity === null) return; + const { provider, model } = identity; applyLiveModelSwitch( { providerName: provider, model }, { @@ -446,12 +445,8 @@ export async function wireSettings( }; const onFavoriteToggle = (id: string): void => { - const sep = id.indexOf(":"); - if (sep <= 0) return; - const ref: ModelRef = { - provider: id.slice(0, sep), - model: id.slice(sep + 1), - }; + const ref = modelOptionRef(id); + if (ref === null) return; void (async () => { let next: Settings | undefined; const result = await services.globalSettingsWriter.mutateAt( @@ -477,12 +472,8 @@ export async function wireSettings( }; const onSetDefault = (id: string): void => { - const sep = id.indexOf(":"); - if (sep <= 0) return; - const ref: ModelRef = { - provider: id.slice(0, sep), - model: id.slice(sep + 1), - }; + const ref = modelOptionRef(id); + if (ref === null) return; void (async () => { let next: Settings | undefined; const result = await services.globalSettingsWriter.mutateAt( diff --git a/src/tui/runner/state.ts b/src/tui/runner/state.ts index 8d132a982..1db1d4680 100644 --- a/src/tui/runner/state.ts +++ b/src/tui/runner/state.ts @@ -15,8 +15,8 @@ import type { InferenceSource, } from "@intx/types/runtime"; import type { Config } from "../../config/index.js"; -import { codexProfileFromProviderName } from "../../config/codex-providers.js"; -import { xaiProfileFromProviderName } from "../../config/xai-providers.js"; +import { peekSourceCredentialSecret } from "../../config/source-credentials.js"; +import { sanitizeDiagnosticText } from "../../diagnostic-sanitize.js"; import type { MCPServerConfig, MCPServerSettingsEntry, @@ -35,6 +35,11 @@ import type { PendingImageAttachment } from "../image-attachments.js"; import type { AgentDeliveryResult } from "../delivery-queue.js"; import type { CompactionLifecycle } from "../../session/compaction-lifecycle.js"; import type { SubmitOutcome } from "./submit.js"; +import { + createCredentialRecoveryState, + type CredentialRecoveryAttempt, + type PendingCredentialRecovery, +} from "./credential-recovery.js"; import type { mountRunnerHost } from "./host.js"; import { EventEmitter } from "node:events"; @@ -228,12 +233,6 @@ export interface RunnerState { liveSource: InferenceSource; liveSources: InferenceSource[]; liveDefaultSource: string; - // The active Codex/xAI source, tracked whenever an OAuth profile source is - // selected so its access token can be refreshed before each send. - activeCodexSource: { profile: string; source: InferenceSource } | undefined; - activeXaiSource: { profile: string; source: InferenceSource } | undefined; - initialCodexProfile: string | undefined; - initialXaiProfile: string | undefined; // MCP servers connected so far, keyed by name so a reconnect after a // failure replaces rather than duplicates the entry. connectedMcpServers: ConnectedMcpServer[]; @@ -257,8 +256,14 @@ export interface RunnerState { stampProvider: { fn: ((id: string | undefined) => void) | undefined }; // Permission-gate persist notices surface through the shell once it exists. approvalPersistNotice: { notify?: (text: string) => void }; + credentialRecovery: ReturnType; + credentialRecoveryAttempts: WeakMap< + ProviderFailureAttempt, + CredentialRecoveryAttempt + >; // Late-wired cross-module callbacks, in original wiring order. + presentCredentialRecovery?: (pending: PendingCredentialRecovery) => void; enqueueAgentDeliver?: ( deliverToLiveAgent: () => void, onSettle?: (result: AgentDeliveryResult) => void, @@ -308,8 +313,18 @@ export interface RunnerState { withFleetPublicationSuspended?: (reset: () => void) => void; } +export function sanitizeRunnerDiagnostic( + state: Pick, + message: string, +): string { + return sanitizeDiagnosticText(message, [ + peekSourceCredentialSecret(state.liveSource.credentialId), + ]); +} + export function recordRunError(state: RunnerState, err: unknown): void { - state.runError = err instanceof Error ? err.message : String(err); + const message = err instanceof Error ? err.message : String(err); + state.runError = sanitizeRunnerDiagnostic(state, message); } /** The live agent; every rebuild swaps the binding this reads. */ @@ -380,10 +395,6 @@ export function createRunnerState(start: TUIStart): RunnerState { liveSource: initialBundle.selected, liveSources: initialBundle.sources, liveDefaultSource: initialBundle.defaultSource, - activeCodexSource: undefined, - activeXaiSource: undefined, - initialCodexProfile: codexProfileFromProviderName(config.providerName), - initialXaiProfile: xaiProfileFromProviderName(config.providerName), connectedMcpServers: start.resumeSeed.mcpServers, configuredMcpEntries: [...config.mcpServerEntries], liveHookConfig: { ...(config.settings?.hooks ?? {}) }, @@ -393,6 +404,8 @@ export function createRunnerState(start: TUIStart): RunnerState { host: undefined, stampProvider: { fn: undefined }, approvalPersistNotice: {}, + credentialRecovery: createCredentialRecoveryState(), + credentialRecoveryAttempts: new WeakMap(), }; // Saved through onboarding's "save anyway" bypass without a passing // connection test — warn now instead of a bare adapter error on first send. diff --git a/src/tui/runner/submit.ts b/src/tui/runner/submit.ts index 725d33782..8310591bc 100644 --- a/src/tui/runner/submit.ts +++ b/src/tui/runner/submit.ts @@ -46,11 +46,14 @@ import { MAILBOX_MAIL_WAKE_PREFIX } from "../../subagent/mailbox-mail-drive.js"; import { hostOf, liveAgent, + recordRunError, runWhileAgentBusy, + sanitizeRunnerDiagnostic, type RunnerServices, type RunnerState, } from "./state.js"; import { LOG_NAMESPACE_ROOT } from "../../branding.js"; +import { buildCredentialRecoveryAlternatives } from "./credential-recovery.js"; const tuiLogger = getLogger([LOG_NAMESPACE_ROOT, "tui"]); @@ -267,15 +270,18 @@ export function createSubmitPath( captureAuthFailure(getTelemetry(), failure); if (!shouldSettleUiAfterSendFailure(failure.kind)) return; if (failure.kind === "abort") return; - state.runError = err instanceof Error ? err.message : String(err); + recordRunError(state, err); if (presentNotice && !providerFailure.presented) { systemNotice( - tuiSendFailureMessage( - err, - failure.kind, - providerFailure.observed, - attempt, - providerFailure.error, + sanitizeRunnerDiagnostic( + state, + tuiSendFailureMessage( + err, + failure.kind, + providerFailure.observed, + attempt, + providerFailure.error, + ), ), ); services.providerFailureAttempts.markPresented(providerFailure); @@ -292,6 +298,11 @@ export function createSubmitPath( ): Promise => { const attempt = live.attemptIdentity(); const providerFailure = services.providerFailureAttempts.begin(attempt); + const recoveryAttempt = state.credentialRecovery.begin( + message, + attempt.providerId, + ); + state.credentialRecoveryAttempts.set(providerFailure, recoveryAttempt); try { await runWhileAgentBusy(state, async () => { const result = await send(message); @@ -316,6 +327,15 @@ export function createSubmitPath( detail: error instanceof Error ? error.message : String(error), }; } finally { + const pending = state.credentialRecovery.settle( + recoveryAttempt, + buildCredentialRecoveryAlternatives( + state.config, + state.sessionId, + recoveryAttempt.failedProvider, + ), + ); + if (pending !== null) state.presentCredentialRecovery?.(pending); services.providerFailureAttempts.sendSettled(providerFailure); } }; diff --git a/src/tui/runtime-bridge.test.ts b/src/tui/runtime-bridge.test.ts index 8457e7b51..f80069133 100644 --- a/src/tui/runtime-bridge.test.ts +++ b/src/tui/runtime-bridge.test.ts @@ -935,7 +935,7 @@ describe("failed sends", () => { }); const safeMessage = - "Codex Provider failed (credential_failure): upstream 401: secret response body. Authentication failed — log in again."; + "Codex Provider failed (credential_failure): upstream 401: secret response body. Authentication failed — run /connect to reconnect the provider profile."; expect( shell.streamLog.filter((row) => row.text === safeMessage), ).toHaveLength(1); diff --git a/src/tui/stream-event-map.test.ts b/src/tui/stream-event-map.test.ts index d5750c284..6cf170af4 100644 --- a/src/tui/stream-event-map.test.ts +++ b/src/tui/stream-event-map.test.ts @@ -632,7 +632,7 @@ describe("inference.error text", () => { expect(out).toEqual([ { type: "assistant", - text: "xai/work Provider failed (credential_failure): HTTP 401. Authentication failed — log in again.", + text: "xai/work Provider failed (credential_failure): HTTP 401. Authentication failed — run /connect to reconnect the provider profile.", }, ]); }); diff --git a/tests/integration/harness.ts b/tests/integration/harness.ts index 975a33cf8..383f0ac33 100644 --- a/tests/integration/harness.ts +++ b/tests/integration/harness.ts @@ -39,7 +39,7 @@ import { import { OPERATOR_ORIGINATED_FLAG } from "../../src/agent/message-provenance.js"; import { readSourceCredentialMaterial, - registerSourceCredential, + registerSourceCredentialRecord, } from "../../src/config/source-credentials.js"; import { createAgentToolset } from "../../src/agent/tools.js"; import { ID_PREFIX } from "../../src/branding.js"; @@ -117,7 +117,10 @@ export async function openIntegrationSession( opts: OpenIntegrationSessionOpts, ): Promise { const harness = setupHarness(); - registerSourceCredential(INTEGRATION_SOURCE.id, INTEGRATION_SECRET); + registerSourceCredentialRecord(INTEGRATION_SOURCE.id, { + provenance: { kind: "api-key" }, + material: { secret: INTEGRATION_SECRET }, + }); const cwd = mkdtempSync(join(tmpdir(), "corbits-integration-cwd-")); const workdir = join(cwd, ".agent-state", "integration-session"); const evidenceArchiveHolder: { current: CompactionArchive | undefined } = { diff --git a/tests/integration/vendored-carry.test.ts b/tests/integration/vendored-carry.test.ts index 4859cdba3..8519ee5e8 100644 --- a/tests/integration/vendored-carry.test.ts +++ b/tests/integration/vendored-carry.test.ts @@ -19,7 +19,7 @@ import { type } from "arktype"; import { ID_PREFIX } from "../../src/branding.js"; import { readSourceCredentialMaterial, - registerSourceCredential, + registerSourceCredentialRecord, } from "../../src/config/source-credentials.js"; import { createPermissionGate } from "../../src/permission/gate.js"; import { createOptimizedContextStore } from "../../src/session/optimized-context-store.js"; @@ -152,7 +152,10 @@ describe("integration — vendored feature carry", () => { }); const storage = await createOptimizedContextStore(workdir); - registerSourceCredential(INTEGRATION_SOURCE.id, "integration-test-key"); + registerSourceCredentialRecord(INTEGRATION_SOURCE.id, { + provenance: { kind: "api-key" }, + material: { secret: "integration-test-key" }, + }); const agent = await createAgent(def, { sources: [INTEGRATION_SOURCE], defaultSource: INTEGRATION_SOURCE.id, diff --git a/tests/unit/exec/runner.test.ts b/tests/unit/exec/runner.test.ts index 264f9382f..e72e6f412 100644 --- a/tests/unit/exec/runner.test.ts +++ b/tests/unit/exec/runner.test.ts @@ -159,7 +159,9 @@ describe("selected provider refresh failures", () => { } catch (err) { expect(formatCaughtError(err)).toBe(rawDiagnostic); const userMessage = execUserFailureMessage(config, err, false); - expect(userMessage).toBe("Authentication failed — log in again."); + expect(userMessage).toBe( + "Authentication failed — run /connect to reconnect the provider profile.", + ); expect(userMessage).not.toContain(rawDiagnostic); } }); @@ -195,7 +197,7 @@ describe("selected provider refresh failures", () => { message: "HTTP 401", }), ).toBe( - "xai/work Provider failed (credential_failure): HTTP 401. Authentication failed — log in again.", + "xai/work Provider failed (credential_failure): HTTP 401. Authentication failed — run /connect to reconnect the provider profile.", ); }); }); diff --git a/tests/unit/inference-response-kind.test.ts b/tests/unit/inference-response-kind.test.ts index b35c08395..b447c94c0 100644 --- a/tests/unit/inference-response-kind.test.ts +++ b/tests/unit/inference-response-kind.test.ts @@ -26,7 +26,7 @@ import { import { CODEX_RESPONSES_PATH } from "../../src/auth/codex/constants.js"; import { readSourceCredentialMaterial, - registerSourceCredential, + registerSourceCredentialRecord, } from "../../src/config/source-credentials.js"; const CODEX_URL = `https://chatgpt.com/backend-api${CODEX_RESPONSES_PATH}`; @@ -94,7 +94,10 @@ async function runCodexTurn( scheduler: createDefaultScheduler(), }; let seq = 0; - registerSourceCredential(CODEX_SOURCE.credentialId, "test-token"); + registerSourceCredentialRecord(CODEX_SOURCE.credentialId, { + provenance: { kind: "api-key" }, + material: { secret: "test-token" }, + }); return collect( runInference({ turns: [userTurn("hi")], diff --git a/tests/unit/summarizer.test.ts b/tests/unit/summarizer.test.ts index d069250cb..1fe4e43d7 100644 --- a/tests/unit/summarizer.test.ts +++ b/tests/unit/summarizer.test.ts @@ -8,7 +8,7 @@ import { DEFAULT_SUMMARIZER_TIMEOUT_MS, } from "../../src/session/summarizer.js"; import type { Telemetry, TelemetryEvent } from "../../src/telemetry/index.js"; -import { registerSourceCredential } from "../../src/config/source-credentials.js"; +import { registerSourceCredentialRecord } from "../../src/config/source-credentials.js"; const source: InferenceSource = { id: "test", @@ -204,7 +204,10 @@ test("summarizer timeout is honoured independently of the director total timeout try { // The stream parks forever; only the summarizer's own timer can end the call. harness.scenario.stall(); - registerSourceCredential("anthropic", "k"); + registerSourceCredentialRecord("anthropic", { + provenance: { kind: "api-key" }, + material: { secret: "k" }, + }); const summarize = createModelSummarizer({ getSource: () => ({ id: "anthropic", diff --git a/tests/unit/tui/runner.test.ts b/tests/unit/tui/runner.test.ts index fd97e037e..1b0a97415 100644 --- a/tests/unit/tui/runner.test.ts +++ b/tests/unit/tui/runner.test.ts @@ -88,11 +88,11 @@ test("TUI send failures prefer an explicitly reported provider", () => { }, ), ).toBe( - "xai/work Provider failed (credential_failure): HTTP 401. Authentication failed — log in again.", + "xai/work Provider failed (credential_failure): HTTP 401. Authentication failed — run /connect to reconnect the provider profile.", ); }); -test("TUI auth failures tell the user to log in again instead of switching models", () => { +test("TUI auth failures tell the user to run /connect instead of switching models", () => { expect( tuiSendFailureMessage( new Error("401 refresh token rejected"), @@ -103,7 +103,9 @@ test("TUI auth failures tell the user to log in again instead of switching model displayLabel: "Codex", }, ), - ).toBe("Authentication failed — log in again."); + ).toBe( + "Authentication failed — run /connect to reconnect the provider profile.", + ); }); test("loadLocalSettingsWriteBase distinguishes absent from unreadable", async () => { diff --git a/vendor/intx-inference/PATCHES.md b/vendor/intx-inference/PATCHES.md index 8397d98ca..3e2caf6bc 100644 --- a/vendor/intx-inference/PATCHES.md +++ b/vendor/intx-inference/PATCHES.md @@ -26,6 +26,20 @@ Re-carryable, "Kill candidate" = Droppable, and "Companion" entries are Re-carryable but ride their primary patch's disposition (they ship out or die with it). +## harness-ts-auth-recovery + +**Promotion candidate.** `auth.ts`, `harness.ts`, and `reactor.ts` atomically +inject credential-derived identity headers with the live bearer secret, shallow +snapshot the complete source once per call, and expose per-call credential +failure ordinal/history to retry policies. A policy may normalize an attempt +error before the harness computes that ordinal/history, retries, or surfaces the +terminal event, and may preserve a classified refresh diagnostic on abort. After +same-source recovery is exhausted, `credential_failure` is terminal and never +enters automatic source failover. + +**Re-carry:** preserve the exact `isCommitting` boundary and immutable call +snapshot. Companion to `runtime-ts-auth-recovery-context` in `@intx/types`. + ## reactor-ts-atomic-approval-acceptance **Promotion candidate.** `reactor.ts` classifies native `approval.granted` / diff --git a/vendor/intx-inference/src/auth.test.ts b/vendor/intx-inference/src/auth.test.ts index d1258f389..c3b0d9091 100644 --- a/vendor/intx-inference/src/auth.test.ts +++ b/vendor/intx-inference/src/auth.test.ts @@ -58,6 +58,30 @@ describe("injectCredentials", () => { expect(out["content-type"]).toBe("application/json"); }); + test("injects bearer and provider identity from one material snapshot", () => { + let reads = 0; + const out = injectCredentials( + { + authorization: BEARER_CREDENTIAL_SENTINEL, + "chatgpt-account-id": "stale-account", + }, + SOURCE, + () => { + reads++; + return { + secret: "fresh-token", + headers: { "chatgpt-account-id": "fresh-account" }, + }; + }, + ); + + expect(out).toEqual({ + authorization: "Bearer fresh-token", + "chatgpt-account-id": "fresh-account", + }); + expect(reads).toBe(1); + }); + test("non-sentinel values pass through unchanged", () => { const out = injectCredentials( { diff --git a/vendor/intx-inference/src/auth.ts b/vendor/intx-inference/src/auth.ts index a5625beb3..bf87ed219 100644 --- a/vendor/intx-inference/src/auth.ts +++ b/vendor/intx-inference/src/auth.ts @@ -49,24 +49,20 @@ export function injectCredentials( source: InferenceSource, readMaterial: CredentialMaterialResolver, ): Record { - // Resolve the source's secret lazily and once: only when a header actually - // carries a sentinel, so a request with no credential sentinel never touches - // the cell, and the fail-closed read (revoked/absent credential) surfaces only - // when the secret is genuinely needed. - let cachedSecret: string | undefined; - const secret = (): string => { - cachedSecret ??= readMaterial(source.credentialId).secret; - return cachedSecret; + let material: ReturnType | undefined; + const resolveMaterial = (): ReturnType => { + material ??= readMaterial(source.credentialId); + return material; }; const result: Record = {}; for (const [name, value] of Object.entries(headers)) { if (value === CREDENTIAL_SENTINEL) { - result[name] = secret(); + result[name] = resolveMaterial().secret; } else if (value === BEARER_CREDENTIAL_SENTINEL) { - result[name] = `Bearer ${secret()}`; + result[name] = `Bearer ${resolveMaterial().secret}`; } else { result[name] = value; } } - return result; + return { ...result, ...material?.headers }; } diff --git a/vendor/intx-inference/src/harness.ts b/vendor/intx-inference/src/harness.ts index 45bf6cbd8..5077d70a9 100644 --- a/vendor/intx-inference/src/harness.ts +++ b/vendor/intx-inference/src/harness.ts @@ -37,7 +37,10 @@ import type { ContentBlock, } from "@intx/types/runtime"; -import type { CredentialMaterialResolver } from "@intx/types"; +import type { + CredentialMaterial, + CredentialMaterialResolver, +} from "@intx/types"; import { getLogger } from "@intx/log"; @@ -253,6 +256,31 @@ const unconfiguredCredentialResolver: CredentialMaterialResolver = ( ); }; +function sanitizeCredentialDiagnostic( + value: unknown, + secrets: ReadonlySet, +): unknown { + if (typeof value === "string") { + let sanitized = value; + for (const secret of secrets) { + if (secret.length > 0) + sanitized = sanitized + .split(secret) + .join("[redacted: configured credential]"); + } + return sanitized; + } + if (Array.isArray(value)) + return value.map((item) => sanitizeCredentialDiagnostic(item, secrets)); + if (value !== null && typeof value === "object") { + const sanitized: Record = {}; + for (const [key, child] of Object.entries(value)) + sanitized[key] = sanitizeCredentialDiagnostic(child, secrets); + return sanitized; + } + return value; +} + /** * Run one fetch lifecycle and yield its events. Ends on the first * `inference.error` or `inference.done`. The outer `runInference` @@ -1575,6 +1603,26 @@ export async function* runInference( const scheduler = opts.deps.scheduler; const startedAtMs = scheduler.now(); const signal = opts.signal; + // Locally patched — see vendor/intx-inference/PATCHES.md#harness-ts-auth-recovery + const callStartSource = Object.freeze({ ...opts.source }); + const credentialFailureHistory: InferenceError[] = []; + const diagnosticSecrets = new Set(); + const captureCredentialMaterial = ( + credentialId: string, + ): CredentialMaterial | undefined => { + const material = opts.readMaterial?.(credentialId); + if (material?.secret !== undefined) diagnosticSecrets.add(material.secret); + return material; + }; + const readCallMaterial: CredentialMaterialResolver | undefined = + opts.readMaterial === undefined + ? undefined + : (credentialId) => { + const material = captureCredentialMaterial(credentialId); + if (material === undefined) + throw new Error(`Unknown inference credential "${credentialId}".`); + return material; + }; for (let attempt = 1; ; attempt++) { // Metadata an attempt emits before it commits (see `isCommitting`): @@ -1587,7 +1635,12 @@ export async function* runInference( // Locally patched — see vendor/intx-inference/PATCHES.md#harness-ts-commitment-boundary-streaming const preCommit: InferenceEvent[] = []; let committed = false; - let failure: { event: InferenceEvent; error: InferenceError } | undefined; + let failure: + | { + event: Extract; + error: InferenceError; + } + | undefined; // Per-attempt private allocator. `runSingleAttempt` allocates a // seq for every event it yields; if the attempt is discarded on @@ -1599,6 +1652,10 @@ export async function* runInference( let attemptSeq = 0; const attemptOpts: InferenceHarnessOptions = { ...opts, + ...(readCallMaterial !== undefined + ? { readMaterial: readCallMaterial } + : {}), + source: callStartSource, nextSeq: () => attemptSeq++, }; for await (const event of runSingleAttempt(attemptOpts)) { @@ -1619,7 +1676,13 @@ export async function* runInference( // Failure after visible output began. The deltas already // delivered cannot be retracted, so retry is off the table: // surface the error on the single live stream and stop. - yield { ...event, seq: opts.nextSeq() }; + yield { + ...(sanitizeCredentialDiagnostic( + event, + diagnosticSecrets, + ) as typeof event), + seq: opts.nextSeq(), + }; return; } failure = { event, error: event.data.error }; @@ -1653,7 +1716,14 @@ export async function* runInference( return; } - const terminalError = failure.error; + let terminalError = sanitizeCredentialDiagnostic( + policy.normalizeError?.(failure.error, callStartSource) ?? failure.error, + diagnosticSecrets, + ) as InferenceError; + const credentialFailureOrdinal = + terminalError.category === "credential_failure" + ? credentialFailureHistory.length + 1 + : 0; // Consult the policy. Sync throws and Promise rejections both // resolve to an abort decision; the original inference.error @@ -1668,6 +1738,11 @@ export async function* runInference( error: terminalError, attempt, elapsedMs: scheduler.now() - startedAtMs, + source: callStartSource, + credentialFailureOrdinal, + credentialFailureHistory: Object.freeze([ + ...credentialFailureHistory, + ]), }), ); } catch (cause) { @@ -1675,6 +1750,26 @@ export async function* runInference( decision = { kind: "abort" }; } + if ( + terminalError.category === "credential_failure" && + opts.readMaterial !== undefined + ) { + try { + captureCredentialMaterial(callStartSource.credentialId); + } catch { + // The policy owns missing-credential handling; history capture must not + // replace its decision with a second resolver error. + } + terminalError = sanitizeCredentialDiagnostic( + terminalError, + diagnosticSecrets, + ) as InferenceError; + } + + if (terminalError.category === "credential_failure") { + credentialFailureHistory.push(terminalError); + } + if (decision.kind === "abort") { // Flush the buffered pre-commit metadata, then the terminal // `inference.error`, all with re-stamped caller-visible seqs, and @@ -1682,7 +1777,34 @@ export async function* runInference( for (const buffered of preCommit) { yield { ...buffered, seq: opts.nextSeq() }; } - yield { ...failure.event, seq: opts.nextSeq() }; + if (decision.error !== undefined) { + yield { + type: "inference.error", + seq: opts.nextSeq(), + data: { + error: sanitizeCredentialDiagnostic( + decision.error, + diagnosticSecrets, + ) as InferenceError, + partial: sanitizeCredentialDiagnostic( + failure.event.data.partial, + diagnosticSecrets, + ) as PartialMessage, + }, + }; + } else { + yield { + type: "inference.error", + seq: opts.nextSeq(), + data: { + error: terminalError, + partial: sanitizeCredentialDiagnostic( + failure.event.data.partial, + diagnosticSecrets, + ) as PartialMessage, + }, + }; + } return; } diff --git a/vendor/intx-inference/src/reactor.test.ts b/vendor/intx-inference/src/reactor.test.ts index 1acc415eb..d7d86203f 100644 --- a/vendor/intx-inference/src/reactor.test.ts +++ b/vendor/intx-inference/src/reactor.test.ts @@ -6386,24 +6386,24 @@ describe("createReactor — source failover", () => { return { ...handle, attemptedSourceIds }; } - test("fails over to the next source on a credential failure", async () => { + test("does not fail over after credential recovery is exhausted", async () => { const { reactor, events, waitFor, attemptedSourceIds } = multiSourceReactor( { sourceIds: ["s0", "s1"], - resultFor: (id) => - id === "s0" - ? { category: "credential_failure", message: "bad key" } - : "done", + resultFor: () => ({ + category: "credential_failure", + message: "bad key", + }), }, ); reactor.start(); reactor.deliver(makeInboundMessage()); await waitFor("reactor.done"); - // s0 failed on a source-specific error -> immediate failover -> s1. - expect(attemptedSourceIds).toEqual(["s0", "s1"]); - const done = getEvent(events, "inference.done"); - expect(done.data.source.sourceId).toBe("s1"); + expect(attemptedSourceIds).toEqual(["s0"]); + expect(getEvent(events, "inference.error").data.error.category).toBe( + "credential_failure", + ); }); test("fails over on quota exhaustion already retried by the harness", async () => { @@ -6490,8 +6490,8 @@ describe("createReactor — source failover", () => { { sourceIds: ["s0", "s1"], resultFor: () => ({ - category: "credential_failure", - message: "bad key", + category: "retryable", + message: "gateway unavailable", }), }, ); @@ -6502,7 +6502,7 @@ describe("createReactor — source failover", () => { // Both sources fail over; the terminal error is surfaced. expect(attemptedSourceIds).toEqual(["s0", "s1"]); expect(getEvent(events, "inference.error").data.error.category).toBe( - "credential_failure", + "retryable", ); }); }); diff --git a/vendor/intx-inference/src/reactor.ts b/vendor/intx-inference/src/reactor.ts index 921666dda..50c769e9c 100644 --- a/vendor/intx-inference/src/reactor.ts +++ b/vendor/intx-inference/src/reactor.ts @@ -934,14 +934,16 @@ export function createReactor(config: ReactorConfig): Reactor { return; } - // Any remaining error (quota, credential, protocol mismatch, - // retryable, timeout) is source-specific. The harness wrapper owns - // mechanical retry and has already exhausted it against this source - // by the time the reactor sees the error, including honoring a - // provider Retry-After for quota, so re-running the same source - // would only retry-compound. Fail over to the next source instead. - // A pacing delay the leaving source asked for must not gate the - // next source. + if (err.category === "credential_failure") { + pendingPacingDelayMs = 0; + enqueue({ type: "inference.error", error: err, partial }); + return; + } + + // Any remaining error (quota, protocol mismatch, retryable, timeout) + // is source-specific. The harness wrapper owns mechanical retry and + // has already exhausted it against this source by the time the reactor + // sees the error. Fail over to the next source instead. pendingPacingDelayMs = 0; if (failOverToNextSource()) { logger.warn`Failing over to next inference source after ${err.category}`; diff --git a/vendor/intx-types/PATCHES.md b/vendor/intx-types/PATCHES.md index e0b1cdafd..e0d50a8d9 100644 --- a/vendor/intx-types/PATCHES.md +++ b/vendor/intx-types/PATCHES.md @@ -16,6 +16,17 @@ usage emission. Upstream deleted `packages/types/src/sidecar-placement.ts`; no entry lived there. Unaffected by upstream's `credentialId` auth-model rewrite. +## runtime-ts-auth-recovery-context + +`mediated-credential.ts` and `runtime.ts` — credential material can atomically +carry provider identity headers with its bearer secret. Retry situations carry +the immutable call-start source plus per-call credential-failure ordinal and +history; retry policies may expose an error normalizer, and abort decisions may +replace the surfaced classified error. + +**Disposition:** Promotion candidate. **Removal path:** upstream equivalent +credential rotation and retry-context contracts, then drop this entry. + ## runtime-ts-audit-store-load-errors `runtime.ts` — `AuditStore` grows `loadErrors(sessionId, signal?)` so a diff --git a/vendor/intx-types/src/mediated-credential.ts b/vendor/intx-types/src/mediated-credential.ts index 63461665f..00e7fdc77 100644 --- a/vendor/intx-types/src/mediated-credential.ts +++ b/vendor/intx-types/src/mediated-credential.ts @@ -21,9 +21,15 @@ // (resolve a credential row, authorize, decrypt) lives on the delivery side and // is never the plugin's decision. -/** The current secret material behind a credential, read fresh at each use. */ +/** + * The current secret material behind a credential, read fresh at each use. + * + * Locally patched — see vendor/intx-types/PATCHES.md#runtime-ts-auth-recovery-context + */ export interface CredentialMaterial { readonly secret: string; + /** Credential-derived provider headers installed atomically with the secret. */ + readonly headers?: Readonly>; } /** diff --git a/vendor/intx-types/src/runtime.ts b/vendor/intx-types/src/runtime.ts index c1957825f..318fefebd 100644 --- a/vendor/intx-types/src/runtime.ts +++ b/vendor/intx-types/src/runtime.ts @@ -2563,7 +2563,7 @@ export function applyInferenceSourceFields( * (INFERENCE.md § Providers › Streaming Harness) */ export type RetryDecision = - | { kind: "abort" } + | { kind: "abort"; error?: InferenceError } | { kind: "retry"; delayMs: number }; /** @@ -2589,6 +2589,12 @@ export type RetrySituation = { * against integer thresholds should `Math.floor` if they need that. */ readonly elapsedMs: number; + /** Immutable shallow snapshot of the source selected when this call began. */ + readonly source?: Readonly; + /** 1-indexed within credential failures only; zero for other categories. */ + readonly credentialFailureOrdinal?: number; + /** Credential failures observed earlier in this call. */ + readonly credentialFailureHistory?: readonly InferenceError[]; }; /** @@ -2604,9 +2610,14 @@ export type RetrySituation = { * * (INFERENCE.md § Providers › Streaming Harness) */ -export type RetryPolicy = ( +export type RetryPolicy = (( situation: RetrySituation, -) => RetryDecision | Promise; +) => RetryDecision | Promise) & { + readonly normalizeError?: ( + error: InferenceError, + source?: Readonly, + ) => InferenceError; +}; /** * Options for a single inference call. Override the defaults from the agent