ai(local): harden Whisper/Kokoro model lifecycle, low-end performance, integrity & Eco-Mode

[qnbs]

Context

ROADMAP.md lists Local AI & Voice hardening as v2.0 Foundation work, including Whisper/Kokoro behavior on lower-end systems, integrity handling and Eco-Mode. PWA-specific local-AI work is already covered by #483/#478, and Ollama connectivity has its own issue history, but there is no cross-surface owner for the bundled/offline Voice/Local-AI runtime lifecycle itself.

The current Core Migration Ledger deliberately keeps AI services out of the initial Wave-2 Rust extraction. This issue must therefore harden the existing product capability without opportunistically turning it into an AI-in-Rust migration.

Goal

Make local speech/AI execution predictable, cancellable, privacy-truthful and resource-bounded across supported current surfaces, with a contract future Qt adapters can consume later.

Capability inventory

Inventory every relevant local execution path, model and artifact at current main, including where applicable:

• Whisper/STT models and worker/runtime path;
• Kokoro/TTS models and worker/runtime path;
• ONNX/WASM/WebGPU/native acceleration paths;
• model download URLs, versions, hashes and cache/storage locations;
• Ollama/local-network routing where it interacts with the same UI but remains a separate runtime;
• fallback/heuristic paths;
• model-manager UI/status and storage cleanup.

For each record:

execution environment
artifact size
minimum/recommended memory
startup/init cost
cancellation support
offline-after-download behavior
integrity verification
cache lifecycle
failure behavior
privacy/routing truth

Model integrity and lifecycle

• Treat model files as versioned artifacts with verifiable integrity where technically feasible.
• Detect incomplete/corrupt downloads rather than repeatedly trying to initialize them.
• Use atomic/fail-safe cache admission so a partial download is not promoted as a valid model.
• Provide explicit retry/remove/redownload behavior.
• Avoid concurrent initialization/download storms via single-flight/admission control.
• Define upgrade/replacement and stale-model cleanup semantics.

Do not silently delete large user-downloaded models merely because initialization fails once.

Low-end / Eco-Mode contract

Define evidence-based Eco-Mode behavior rather than an ambiguous label. Candidate controls may include, where supported:

• smaller model preference;
• reduced concurrency;
• deferred/prevented background preloading;
• lower memory/cache pressure;
• CPU/GPU backend preference based on measured capability;
• reduced speculative/background AI work.

Eco-Mode must not alter manuscript/project correctness. It may reduce AI quality/performance features explicitly, but autosave/editor interaction must remain responsive.

Responsiveness and resource safety

Measure representative low/mid/high classes for:

• first model initialization;
• warm initialization;
• peak/resident memory;
• STT/TTS latency/throughput;
• cancellation latency;
• long-session memory slope;
• editor input/autosave latency while inference runs;
• simultaneous model request behavior;
• background-tab/window suspend/resume where applicable.

Prevent local inference from starving save/recovery operations or making the editor unusable.

Failure / cancellation semantics

Every long-running local model operation must have a deterministic outcome for:

USER_CANCELLED
TIMEOUT
MODEL_MISSING
MODEL_CORRUPT
MODEL_LOAD_FAILED
RESOURCE_LIMIT
BACKEND_UNAVAILABLE
UNSUPPORTED_CAPABILITY

Do not expose raw worker/ONNX/native exception strings as user guidance. Coordinate portable structured diagnostic semantics with #551 while keeping manuscript/prompt/audio content out of logs by default.

Privacy/routing truth

The UI must clearly distinguish:

• browser/device-local model execution;
• local-network Ollama execution;
• cloud-provider execution;
• any WorldScript-hosted proxy route.

Coordinate user-facing security/privacy wording with #549 and PWA-specific routing/capability acceptance with #483/#478.

Cross-surface strategy

• PWA/browser-specific model/cache/WebGPU acceptance remains owned by pwa(ai): turn WebGPU/worker local inference into a zero-install offline AI differentiator #483/pwa(first-class): harden offline reliability, lifecycle, performance, install/update UX & browser-native AI #478.
• Transitional desktop may reuse current web/native capability where supported.
• Future Qt must consume an admitted capability boundary rather than copy browser-specific lifecycle assumptions into QML.
• AI services remain out of the initial Core migration unless a separate later admission decision changes that.

Regression matrix

At minimum cover:

• first download/init;
• warm offline init after valid download;
• interrupted/corrupt download;
• hash/integrity mismatch where supported;
• two simultaneous init requests;
• cancel during download/load/inference;
• timeout;
• low-memory/resource rejection;
• Eco-Mode on/off;
• backend unavailable/fallback;
• model removal/redownload;
• long-session repeated inference;
• editor/autosave coexistence;
• truthful local/cloud routing labels.

Acceptance criteria

• Current local Voice/AI capability and model inventory is explicit.
• Model download/admission/integrity lifecycle is deterministic and recoverable.
• Concurrent initialization/download storms are prevented.
• Cancellation/timeout/resource-limit failures are typed and user-actionable.
• Eco-Mode has a concrete, tested resource policy.
• Low-end performance/memory budgets exist and protect editor/autosave responsiveness.
• Offline-after-download behavior is proven where promised.
• UI privacy/routing claims match actual execution paths.
• PWA-specific work is reconciled with pwa(ai): turn WebGPU/worker local inference into a zero-install offline AI differentiator #483 rather than duplicated.
• Future Qt consumption does not require browser implementation leakage.

Non-goals

• porting all AI services into Rust;
• changing cloud-provider product policy;
• implementing collaboration/local-first architecture;
• inventing telemetry that uploads manuscript/audio content;
• coupling this work to the current S2 bootstrap convergence.

[qnbs]

Product/UX exploration for this topic now also lives in Discussion #632: #632. This Issue remains the implementation/acceptance tracker; the Discussion does not change priority, roadmap admission, acceptance criteria, or execution sequencing by itself.

[qnbs]

v1.28.7 release-gate relationship

#743 is now the durable AI / Local-AI pre-release PASS/FAIL tracker.

#556 remains the implementation owner for Whisper/Kokoro/local-AI lifecycle, integrity, low-end/resource behavior and Eco-Mode. The release-critical subset before v1.28.7 is explicitly captured in #743: current runtime/model inventory, artifact/version truth, corrupt/incomplete download handling, bounded init/download concurrency, cancellation/timeout semantics, resource/backend failure behavior, offline-after-download truth, concrete Eco-Mode behavior, editor/autosave coexistence, and truthful on-device vs local-network vs cloud labeling.

The whole long-term #556 scope is not automatically a release blocker. Promote any newly discovered P0/P1 or release-critical P2 under #743 (privacy/local-only violation, destructive recovery, unusable advertised runtime/default, serious resource exhaustion/crash, materially false execution/offline claim). Keep non-critical future hardening owned here without expanding unrelated current PRs.