aqe fallback chain is never credential-checked — ak writes and reports dead rungs as healthy

[pacphi]

Summary

ak will happily write — and then report as healthy — an agentic-qe fallback chain whose rungs cannot execute, because nothing anywhere checks that a chain entry's provider has a usable credential. On a real machine this produces a chain with exactly one live rung and a silent dead end underneath it.

A second, related gap: openrouter is absent from API_PROVIDERS, so the one metered provider that is credentialed on this machine is invisible to every ak surface — while openai, which is not credentialed, is displayed as a configured fallback.

Observed

Machine state (ak x provider status, verbatim):

agentic-qe LLM provider  (AQE_LLM_PROVIDER)
  aqe default (unset)      supported: claude-code, claude, openai, gemini, ...
  fallback chain: claude-code(claude-opus-4-8) → openai(gpt-5.6) · .agentic-qe/llm-config.json

ruflo LLM API providers  (ruflo router; keys read from env)
  anthropic  no key       not configured
  openai     no key       not configured
  google     no key       not configured
  ollama     local        not configured

✓ provider config reflects installed CLIs

Environment (redacted, filtered from env):

OPENROUTER_API_KEY=<set>
# no ANTHROPIC_API_KEY, no OPENAI_API_KEY, no GEMINI_API_KEY / GOOGLE_API_KEY

So:

Rung	Provider	Model	Credential	Actually usable?
priority 100	claude-code	claude-opus-4-8	oauth / subscription	✅
priority 90	openai	gpt-5.6	OPENAI_API_KEY — absent	❌ dead rung
—	openrouter	(not in chain)	OPENROUTER_API_KEY — present	invisible to ak

.agentic-qe/llm-config.json writes "providers": { "openai": { "enabled": true } } and a priority-90 chain entry for it. When claude-code exhausts its maxAttempts: 2, the router fails over to a provider with no key. The two ak surfaces that could have caught this both report green:

• ak x provider status prints ✓ provider config reflects installed CLIs — it validates the host axis, not chain credentials.
• ak status's providers row compares only chain order against disk (src/commands/status.mjs:336-349), so an all-dead chain in the right order reads ok.

Root cause

Three separate places validate a chain entry, and none of them touches credentials:

1. applyAqeRouter (src/lib/providers.mjs:287-302) filters on AQE_PROVIDER_TYPES.includes(e.provider) and warns on empty models — a type check and a shape check. No credential check.
2. ak x provider pick (src/commands/x/provider.mjs:329) validates chain providers at prompt time on the same type-only basis, so the user is never told the rung they just typed is inert.
3. ak status (src/commands/status.mjs:336-349) computes routerDrift from diskOrder !== chain.map(e => e.provider).join('→'). Order-only.

The credential probe that does exist — detectProviders() (src/lib/providers.mjs:155-161) — is scoped to API_PROVIDERS, which is the ruflo router axis:

export const API_PROVIDERS = [
  { id: 'anthropic', keyEnv: ['ANTHROPIC_API_KEY'] },
  { id: 'openai', keyEnv: ['OPENAI_API_KEY'] },
  { id: 'google', keyEnv: ['GOOGLE_API_KEY', 'GEMINI_API_KEY'] },
  { id: 'ollama', keyEnv: [] },
];

AQE_PROVIDER_TYPES (src/lib/providers.mjs:63-66) has ten members; API_PROVIDERS has four, and the two lists were never reconciled. openrouter, azure-openai, bedrock, cognitum, gemini, claude-code, onnx have no keyEnv mapping anywhere in the codebase, so a chain built from any of them is unverifiable by construction — and openrouter in particular can never show key present even when it is the only credential on the box.

This is arguably the most user-visible instance: PROVIDER_MODEL_CATALOG (src/lib/routing.mjs:66-71) already curates GLM models specifically for the openrouter fallback path, and its own comment says keys "live in the env (e.g. OPENROUTER_API_KEY)" — so the feature assumes a credential the detection layer cannot see.

Impact

• A documented, ak-managed failover path is inert, and the failure only surfaces at QE-run time as a provider error — far from the config that caused it.
• ak status reporting ok on a chain with one live rung is an incorrect health signal, which is the specific thing the providers row exists to prevent.
• The suggested default makes this the likely state, not an edge case: AQE_FALLBACK_CODEX_SUGGESTION = 'claude-code:claude-opus-5; openai:gpt-5.6' (src/lib/providers.mjs:219) is offered whenever codex is an enabled host. Codex-the-host authenticates via oauth, so a user who enables codex reasonably accepts the suggestion — and gets an openai rung requiring an API key they were never asked for. The suggestion's own comment notes codex models are reached "via the openai provider type", which is true of the model IDs but not of the auth.

Proposal

1. Give every aqe provider type a credential descriptor. Extend the API_PROVIDERS shape (or add a parallel AQE_PROVIDER_CREDENTIALS map) covering all of AQE_PROVIDER_TYPES:

Provider	Credential	Billing
claude-code	host oauth (hostAuthState('claude'))	subscription
claude	ANTHROPIC_API_KEY	metered
openai	OPENAI_API_KEY	metered
gemini	GEMINI_API_KEY | GOOGLE_API_KEY	metered
openrouter	OPENROUTER_API_KEY	metered
azure-openai	AZURE_OPENAI_API_KEY (+ endpoint)	metered
bedrock	AWS credential chain	metered
cognitum	COGNITUM_API_KEY	metered
ollama / onnx	local, no key	$0

Exact env names for azure/bedrock/cognitum should be grounded against agentic-qe's provider implementations before landing, not guessed — same discipline as the AQE_PROVIDER_TYPES comment already applies.

2. Warn at write time. In applyAqeRouter, alongside the existing ⚠ no models for: detail, emit ⚠ no credential for: openai for any rung whose credential is absent. Do not refuse to write — a user may export the key later, and silently dropping a rung would be worse than a warning.

3. Warn at pick time. ak x provider pick should say, at the moment of entry, openai: no OPENAI_API_KEY in env — this rung will fail over into nothing, and offer the credentialed alternatives it can actually see.

4. Extend ak status's drift check from order-only to viability. Add a warn row: aqe chain: 1/2 rungs have credentials (openai: no key). Keep it warn, not fail — the primary rung still works.

5. Add openrouter to whatever ak x provider status renders as the provider table, so a credentialed provider is never invisible. Given PROVIDER_MODEL_CATALOG already curates GLM models for it, it should also be offerable as a chain rung in pick.

Acceptance criteria

• Every member of AQE_PROVIDER_TYPES has a credential descriptor; a unit test asserts the two lists cannot diverge.
• applyAqeRouter returns a detail string naming credential-less rungs; covered by a test with a stubbed env.
• ak x provider pick warns interactively on a credential-less rung.
• ak status emits a warn providers row when any chain rung lacks a credential; ok only when every rung is viable.
• ak x provider status shows openrouter with key present when OPENROUTER_API_KEY is set.
• Reproduction above (claude-code + openai chain, no OPENAI_API_KEY) produces a warning on all three surfaces.

Notes

Per agentic-qe's documented HybridRouter behavior, it auto-enables any provider with a key in the env, so openrouter may already be reachable at runtime on this machine. That is exactly the problem this issue describes from the other side: the effective routing does not match ak's declared chain, and ak cannot tell the user which is which. The stated point of writing .agentic-qe/llm-config.json from kit.json is deterministic ordering rather than env auto-enable — that guarantee doesn't hold while credentials go unchecked.

Related: #36 (multi-LLM provider UX).

[pacphi]

Companion: #55 — same class of failure (a green ak status providers row that means "matches disk" rather than "is usable/current"), different axis. Worth fixing the two report paths together.