Origin / user request
WorldScript Studio's AI subsystem needs a comprehensive current-state requalification.
Two related problems need to be addressed together:
- Model catalogues, defaults, fallbacks and provider-specific hard-coded model IDs have accumulated over time and are no longer guaranteed to represent the best currently available supported models.
- The application's AI functionality has grown substantially, but the complete set of user-facing AI workflows has not yet been systematically exercised end-to-end against the real provider/runtime paths.
This issue owns a full AI model/provider refresh plus functional qualification of the complete AI product surface.
The goal is not merely to replace a few model strings. The goal is to establish:
CURRENT PROVIDER TRUTH
↓
CANONICAL MODEL/CAPABILITY AUTHORITY
↓
VETTED DEFAULTS
↓
CORRECT PROVIDER ADAPTERS
↓
REAL FUNCTIONAL QUALIFICATION
↓
MAINTAINABLE UPDATE / DEPRECATION POLICY
Why this is needed now
Current repository evidence already demonstrates model-authority drift.
Examples include:
AiModel / settings contain comparatively recent Gemini/OpenAI/Anthropic/Grok model families;
- the canonical Gemini text fallback is still independently hard-coded in more than one runtime path;
DefaultInferenceGateway.modelList() still exposes an older gemini-2.0-flash entry;
- OpenRouter has its own preferred-model/default/fallback authority;
- model choices appear in Settings/UI, storage normalization, inference routing, provider adapters, tests, documentation and help copy;
- local inference maintains additional independent WebLLM/ONNX/Transformers model catalogues.
This is concrete evidence that repository comments such as "current catalog" or "latest generation" cannot themselves be treated as current upstream truth.
All provider/model claims must therefore be re-derived from authoritative upstream sources at implementation time.
Do not assume that model names presently labelled "current" in source code are actually current.
1. Live upstream provider inventory
Before changing code, perform a live current-state investigation of every supported AI backend.
WorldScript currently models these provider classes:
gemini
openai
anthropic
grok
ollama
openrouter
webllm
onnx
transformers
Also account for local OpenAI-compatible runtime presets where applicable, including:
Ollama
LM Studio
vLLM
custom OpenAI-compatible endpoint
For each externally managed provider, inspect the official provider documentation/API at execution time.
Record at minimum:
| Field |
Required |
| exact model ID |
yes |
| human display name |
yes |
| availability state |
stable / preview / experimental / deprecated / shutdown |
| text generation |
yes/no |
| streaming |
yes/no |
| structured/JSON output |
yes/no |
| image input |
yes/no |
| image generation |
yes/no |
| tool/function calling |
yes/no where relevant |
| reasoning support |
yes/no/variant |
| context limit |
where authoritative |
| output-token limit |
where relevant |
| provider-specific parameter restrictions |
yes |
| API/SDK compatibility |
yes |
| pricing/free-tier status |
when surfaced in WorldScript UX |
| deprecation/shutdown date |
where published |
| recommended replacement |
where published |
Prefer official documentation, model-list APIs and official deprecation notices over blog posts, third-party model lists or repository comments.
For OpenRouter, verify current OpenRouter metadata rather than assuming a historical :free model remains free or available.
2. Separate "newest" from "best default"
Do not mechanically make every newest preview/experimental model the application default.
Establish explicit model lifecycle categories such as:
RECOMMENDED_DEFAULT
CURRENT_SUPPORTED
SPECIALIZED
PREVIEW_OPT_IN
LEGACY_COMPAT
DEPRECATED
REMOVED_UPSTREAM
UNKNOWN_STORED_VALUE
Selection of the WorldScript default should consider:
quality
stability
latency
availability
cost
rate limits
context window
structured-output reliability
streaming support
provider compatibility
creative-writing quality
A newer experimental model may be listed as an opt-in choice without replacing a proven stable production default.
Conversely, a deprecated/shutdown model must not remain an active default merely because it still exists in persisted types.
3. Inventory every model authority in the repository
Perform a repository-wide model-string and provider-authority audit.
At minimum inspect:
types.ts / AiModel / AIProvider;
features/settings/settingsSlice.ts;
components/settings/AiSections.tsx;
components/settings/OpenRouterSection.tsx;
services/geminiService.ts;
services/aiProviderService.ts;
services/ai/providerFactory.ts;
services/ai/inferenceGateway.ts;
services/ai/worldScriptCompletionFetch.ts;
services/ai/aiModeService.ts;
services/ai/aiConstants.ts;
- hybrid fallback/routing;
- provider connection testing;
- IDB/settings normalization;
- filesystem/cloud settings persistence;
- per-project AI overrides;
- ProForge AI configuration;
- local AI model registries;
- WebLLM supported models;
- ONNX supported models;
- Transformers.js model assumptions;
- image-generation model IDs;
- documentation/help/locales;
- tests and fixtures;
- historical compatibility paths.
Search for all literal model IDs, not only known defaults.
Classify each occurrence as:
CANONICAL_AUTHORITY
DERIVED_VIEW
PERSISTED_LEGACY_COMPAT
TEST_FIXTURE
DOCUMENTATION
STALE_DUPLICATE_AUTHORITY
REMOVE
A model ID should not silently acquire multiple independent "default" authorities.
4. Establish a canonical model registry
Where practical, replace scattered current-model knowledge with one explicit capability-aware authority.
A model entry should be able to represent properties conceptually equivalent to:
provider
modelId
displayName
lifecycle
capabilities
stability
recommendedRoles
requiresApiKey
localOrCloud
supportsStreaming
supportsStructuredOutput
supportsImageInput
supportsImageGeneration
supportsReasoning
contextLimit
outputLimit
deprecation
replacementModel
Do not over-engineer this into a remote model-management platform.
The immediate objective is one auditable source from which Settings model choices, runtime validation, provider defaults, capability filters, model labels and fallback selection can derive consistently.
Provider APIs may still supply dynamic availability information, but WorldScript should retain an explicit curated admission layer rather than automatically trusting every remotely advertised model.
5. Remove stale duplicated defaults
After establishing authority, reconcile every fallback/default.
Patterns such as:
model?.startsWith('gemini-') ? model : '<hard-coded-model>'
must not independently drift from Settings, AiModel, inference gateways or model selectors.
The same applies to OpenAI defaults, Claude defaults, Grok defaults, OpenRouter fallback models, image-generation defaults, per-project preset defaults and local inference defaults.
Prefer shared provider-specific recommended defaults sourced from the canonical catalogue.
6. Capability-aware routing
Do not assume that every model from the same provider supports every WorldScript operation.
Build or verify capability admission for operations including:
TEXT
STREAMING_TEXT
STRUCTURED_JSON
IMAGE_INPUT
IMAGE_GENERATION
REASONING
TOOLS / FUNCTION CALLING
EMBEDDINGS / RAG
Only include capabilities WorldScript actually uses.
If a feature requires structured output, do not route it to a model merely because that model can generate ordinary prose.
If image generation is Gemini-specific today, make that truth explicit instead of giving other provider selections the appearance that image generation will work universally.
7. Provider adapter requalification
Exercise each currently supported provider adapter against current APIs/SDKs.
Gemini
Verify current text model, current image-generation model, structured JSON, streaming where used, current SDK semantics, reasoning/thinking configuration where applicable and deprecation removal.
OpenAI
Verify current supported model IDs, reasoning model compatibility, parameter differences between model families, streaming, structured output and custom/OpenAI-compatible endpoint behavior.
Do not assume every new OpenAI model accepts exactly the same temperature/token/reasoning parameters.
Anthropic
Verify current native Claude support, exact model IDs, request structure, output extraction, cancellation, token limits and provider-specific options.
xAI / Grok
Verify current exact IDs and OpenAI-compatible/native behavior actually used by WorldScript.
OpenRouter
Verify current endpoint contract, model IDs, free/paid state, account-credit UX, fallback behavior, routing/error metadata and rate-limit handling.
A historical :free identifier must not be presented as guaranteed free without current evidence.
Local/OpenAI-compatible
Verify Ollama, LM Studio, vLLM and custom endpoints separately where WorldScript claims support.
Do not equate "OpenAI-compatible" with complete behavioral parity automatically.
8. Local browser AI model requalification
Audit:
WebLLM
ONNX Runtime Web
Transformers.js
For each bundled/advertised local model determine whether the model still exists, the artifact still downloads, the current library supports it, memory footprint, browser compatibility, WebGPU/WASM requirements, approximate download size, practical minimum memory, context limits, and whether it actually performs the intended writing task acceptably.
Remove or clearly classify models that are technically loadable but no longer useful as realistic user-facing defaults.
Do not replace lightweight local defaults with models that make the application unusable on ordinary hardware solely because they are newer.
9. Persisted-model migration and compatibility
Existing installations may contain historical model IDs.
Do not blindly rewrite valid user preferences.
Classify persisted states:
CURRENT_SUPPORTED
DEPRECATED_BUT_AVAILABLE
REMOVED_UPSTREAM
UNKNOWN
MALFORMED
Required behavior:
- current supported explicit selection → preserve;
- deprecated but still usable → preserve or warn according to policy;
- removed/shutdown → offer/use a defined safe replacement;
- unknown future/custom model → preserve where the provider supports arbitrary IDs rather than destructively normalizing it away;
- invalid combinations → fail with actionable UX rather than silently routing to an unrelated provider.
Document all automatic migration rules.
10. Full inventory of AI product features
The second half of this issue is equally important:
Every actual AI feature exposed by WorldScript must be exercised and qualified.
First generate an authoritative inventory from all generateText, generateJson, generateImage, streaming/inference-gateway and equivalent call sites.
Known examples already include:
- Writer/completion workflows;
- manuscript continuation/generation;
- synopsis generation;
- character generation and field regeneration;
- world generation and field regeneration;
- Critic analysis;
- Consistency Checker;
- outline/story planning flows;
- scene-related generation;
- character interviews where AI-backed;
- scene visualization;
- character portraits;
- world images;
- ProForge pipeline agents;
- AI API/internal orchestration;
- plugin
ai.invoke capability where enabled;
- model/provider connection tests;
- RAG/local-routing flows;
- per-project AI presets;
- hybrid fallback paths.
Do not assume this list is complete. Derive the final inventory directly from the current repository.
11. Build an AI feature × provider × capability matrix
Produce a matrix such as:
| Feature |
Text |
JSON |
Image |
Streaming |
Gemini |
OpenAI |
Claude |
Grok |
OpenRouter |
Local |
| Continue writing |
✓ |
|
|
✓ |
|
|
|
|
|
|
| Critic |
✓ |
|
|
|
|
|
|
|
|
|
| Consistency |
✓ |
structured/parsing |
|
|
|
|
|
|
|
|
| Character generation |
|
✓/text |
|
|
|
|
|
|
|
|
| Character portrait |
|
|
✓ |
|
|
|
|
|
|
|
| … |
|
|
|
|
|
|
|
|
|
|
Every cell must be classified as one of:
SUPPORTED_AND_QUALIFIED
SUPPORTED_NOT_YET_QUALIFIED
UNSUPPORTED_BY_PROVIDER
BLOCKED_BY_POLICY
LOCAL_ONLY
CLOUD_ONLY
NOT_APPLICABLE
Avoid pretending that every provider supports every feature.
12. Real functional smoke qualification
Unit mocks are necessary but insufficient for this work.
Establish a manual/opt-in real-provider qualification suite that can exercise actual APIs without putting provider secrets into the normal repository or mandatory public CI.
Do not add production API keys to GitHub.
For each supported provider with available maintainer credentials, verify representative real requests.
At minimum test:
Text
- simple generation;
- long-form writing continuation;
- cancellation;
- streaming where supported;
- non-empty result;
- Unicode/non-English text.
Structured output
- valid expected schema;
- malformed-model-output recovery;
- parser behavior;
- retry/fallback behavior.
Images
Where supported:
- character portrait;
- world image;
- scene visualization;
- valid returned artifact;
- error/cancellation path.
Local inference
- model initialization/download;
- successful inference;
- offline behavior after model availability;
- unsupported browser/runtime behavior;
- cancellation;
- memory/resource cleanup.
13. Test quality without brittle LLM assertions
Do not write tests expecting exact creative prose from live models.
Live qualification should assert stable properties such as request accepted, response non-empty, schema valid, language broadly correct, required keys present, image payload valid, no provider error, cancellation works and latency bounded enough for smoke qualification.
Mocked deterministic CI contract tests should cover exact adapter behavior separately.
14. Connection-test correctness
Audit every "Test connection" UI.
A green connection test must prove the configured provider/model can actually perform the minimum operation WorldScript will ask from it.
Do not report success merely because an API endpoint responds, a key has valid syntax or a provider account exists.
Differentiate errors such as:
NO_API_KEY
INVALID_KEY
MODEL_NOT_FOUND
MODEL_DEPRECATED
MODEL_ACCESS_DENIED
INSUFFICIENT_CREDITS
RATE_LIMITED
NETWORK_UNAVAILABLE
LOCAL_BACKEND_OFFLINE
CAPABILITY_UNSUPPORTED
PROVIDER_ERROR
Expose actionable UI without leaking sensitive provider responses.
15. Fallback-chain qualification
Audit hybrid fallback semantics.
Verify primary provider first, configured order respected, no duplicate provider attempts, unsupported capabilities skipped intentionally, auth failure does not create dangerous/unexpected cross-provider behavior, rate limits/transient errors follow defined policy, local-only/privacy modes cannot silently fall back to cloud and cloud fallback never bypasses explicit user privacy policy.
A fallback must not silently change from an image-capable model to a text-only provider and then surface an opaque failure.
16. AI mode qualification
Exercise all modes:
For each mode verify routing authority, model selection, settings UI, cold-start restoration, provider fallback, privacy policy, local/cloud boundaries and error messaging.
No mode should exist only as UI/state while behaving identically to another mode in runtime unless explicitly documented.
17. BYOK / secret safety
Preserve current provider-key security architecture.
During this work:
- never commit API keys;
- never include keys in logs;
- never persist provider keys in general Redux/project serialization;
- retain dedicated encrypted key-store authority;
- never expose secrets to plugins;
- redact provider errors where they may contain account/request metadata;
- do not weaken CSP/storage/privacy policy merely to support a provider.
Real qualification credentials must remain external to the repository.
18. Prompt / model compatibility
Model upgrades can change behavior even when the transport works.
Run representative prompt qualification for major WorldScript use cases.
Inspect system/user role assumptions, JSON-format instructions, prompt length, thinking/reasoning controls, stop sequences, temperature/top-p support, max-token semantics, multilingual prompts and creative-writing behavior.
Do not automatically carry provider-specific parameters to a model family that rejects or ignores them.
19. Defaults by workload, not one global model everywhere
Evaluate whether one default model is appropriate for all operations.
Potential roles include:
GENERAL_WRITING_DEFAULT
FAST_LOW_COST_DEFAULT
HIGH_QUALITY_REASONING
STRUCTURED_OUTPUT
IMAGE_GENERATION
LOCAL_LIGHTWEIGHT
OPENROUTER_FREE
If different workloads legitimately need different models, make that mapping explicit rather than scattering hidden exceptions.
Do not add complexity unless qualification evidence justifies it.
20. UI model catalogue correctness
Settings must show truthful current model information.
Verify provider grouping, display names, default selection, stable vs preview badges where useful, local/cloud indicator, free vs paid claims, unavailable/deprecated state, selected legacy value, capability mismatch and model access errors.
A removed model should not remain presented as a normal recommended choice.
21. Documentation and localization
After implementation, reconcile README, Help, AI/provider documentation, settings copy, provider counts, examples, model names, screenshots if model names are visible and all locale bundles through the normal i18n pipeline.
There is already evidence that provider-count/model-copy can drift between locales. Treat generated locale bundles according to repository authority rather than manually editing derived output.
Avoid prose such as "latest model" unless there is a maintainable definition or source.
22. Prevent the catalogue becoming stale again
Establish a bounded maintenance policy.
Prefer:
official upstream verification
↓
curated registry update
↓
focused provider contract tests
↓
real qualification
↓
release
Consider a scheduled/advisory model-health check for providers exposing stable machine-readable model metadata.
It must not automatically rewrite production defaults, make external provider availability a flaky mandatory PR gate, or automatically trust every remotely advertised model.
It may detect deleted/deprecated configured model IDs where reliably possible and produce actionable maintenance evidence.
23. Deprecation policy
Define what happens when a provider announces model retirement.
Suggested lifecycle:
CURRENT
↓
DEPRECATION ANNOUNCED
↓
REPLACEMENT QUALIFIED
↓
DEFAULT MOVED
↓
LEGACY SELECTION WARNED
↓
UPSTREAM SHUTDOWN
↓
REMOVED FROM NORMAL PICKER
Persisted projects/settings must not become impossible to load because an AI model disappeared.
Model selection is configuration, not project-data validity.
24. Performance / cost qualification
For recommended cloud defaults capture representative latency, output quality, token usage, context capability, rate limits and approximate cost class.
This does not require a full AI benchmark laboratory. It does require enough evidence to avoid selecting a dramatically slower or substantially more expensive default merely because it is newer.
For local models capture download size, startup/load time, RAM/VRAM requirements, representative generation speed and browser/runtime support.
25. Privacy-safe qualification evidence
Record a reusable AI qualification report without storing user manuscripts or secret prompts.
For each run record only what is necessary, for example:
provider
model
feature
result
capability
duration
error category
environment
date
application SHA
Do not log API keys, manuscript prose, private prompts/results, account identifiers, provider tokens or private filesystem paths.
Synthetic qualification fixtures should be used where possible.
26. CI versus real-provider testing
Maintain a clear separation.
Mandatory CI
Use deterministic mocks/fakes/contract fixtures for request construction, response parsing, model routing, fallback behavior, capability admission, persistence, cancellation and error classification.
Real provider qualification
Use controlled manual/opt-in runs for actual API availability, current model ID validity, provider behavior, real streaming, real structured output and real image generation.
Do not make ordinary PR CI depend on billable external AI APIs.
27. Architecture boundary
This issue should improve the current React/PWA/Tauri AI product without creating Tauri-only AI architecture.
Keep reusable semantics renderer-neutral where practical:
provider identity
model identity
capabilities
routing
policy
request/response contracts
The future Qt client should be able to consume the same provider/model authority through the intended shared boundaries rather than reimplementing another catalogue.
Do not start Qt implementation from this issue.
28. Implementation slicing
This issue is broad enough that implementation will likely require multiple causal PRs.
Prefer slices such as:
A. live inventory + canonical model authority
B. provider/default/catalogue reconciliation
C. cloud provider contract corrections
D. local model requalification
E. application-wide AI functional qualification
F. docs/UX/maintenance policy
Re-derive the exact split after inspection.
Do not submit one huge mixed PR if the work naturally separates.
Each PR must remain individually reviewable and reversible.
Acceptance criteria
Non-goals
- automatically adopting every new preview/experimental model;
- claiming the newest model is necessarily the best WorldScript default;
- putting paid external AI calls into mandatory PR CI;
- committing maintainer/provider credentials;
- deleting old persisted model values without compatibility handling;
- automatically trusting arbitrary models returned by a remote catalogue;
- rewriting the entire AI architecture unnecessarily;
- making every provider support every AI feature artificially;
- weakening privacy/local-only policy for fallback convenience;
- starting Qt implementation;
- benchmarking every model on the market;
- turning WorldScript into a generic provider/model marketplace.
Priority / disposition
P1
AI is a primary advertised WorldScript capability. The repository already contains demonstrable model-authority drift, and the complete AI surface has not yet received systematic real-provider functional qualification.
This is therefore more than routine model-version housekeeping: stale or retired model IDs, unsupported parameter combinations or unqualified provider paths can make core advertised functionality fail at runtime despite ordinary mocked CI remaining green.
Origin / user request
WorldScript Studio's AI subsystem needs a comprehensive current-state requalification.
Two related problems need to be addressed together:
This issue owns a full AI model/provider refresh plus functional qualification of the complete AI product surface.
The goal is not merely to replace a few model strings. The goal is to establish:
Why this is needed now
Current repository evidence already demonstrates model-authority drift.
Examples include:
AiModel/ settings contain comparatively recent Gemini/OpenAI/Anthropic/Grok model families;DefaultInferenceGateway.modelList()still exposes an oldergemini-2.0-flashentry;This is concrete evidence that repository comments such as "current catalog" or "latest generation" cannot themselves be treated as current upstream truth.
All provider/model claims must therefore be re-derived from authoritative upstream sources at implementation time.
Do not assume that model names presently labelled "current" in source code are actually current.
1. Live upstream provider inventory
Before changing code, perform a live current-state investigation of every supported AI backend.
WorldScript currently models these provider classes:
Also account for local OpenAI-compatible runtime presets where applicable, including:
For each externally managed provider, inspect the official provider documentation/API at execution time.
Record at minimum:
Prefer official documentation, model-list APIs and official deprecation notices over blog posts, third-party model lists or repository comments.
For OpenRouter, verify current OpenRouter metadata rather than assuming a historical
:freemodel remains free or available.2. Separate "newest" from "best default"
Do not mechanically make every newest preview/experimental model the application default.
Establish explicit model lifecycle categories such as:
Selection of the WorldScript default should consider:
A newer experimental model may be listed as an opt-in choice without replacing a proven stable production default.
Conversely, a deprecated/shutdown model must not remain an active default merely because it still exists in persisted types.
3. Inventory every model authority in the repository
Perform a repository-wide model-string and provider-authority audit.
At minimum inspect:
types.ts/AiModel/AIProvider;features/settings/settingsSlice.ts;components/settings/AiSections.tsx;components/settings/OpenRouterSection.tsx;services/geminiService.ts;services/aiProviderService.ts;services/ai/providerFactory.ts;services/ai/inferenceGateway.ts;services/ai/worldScriptCompletionFetch.ts;services/ai/aiModeService.ts;services/ai/aiConstants.ts;Search for all literal model IDs, not only known defaults.
Classify each occurrence as:
A model ID should not silently acquire multiple independent "default" authorities.
4. Establish a canonical model registry
Where practical, replace scattered current-model knowledge with one explicit capability-aware authority.
A model entry should be able to represent properties conceptually equivalent to:
Do not over-engineer this into a remote model-management platform.
The immediate objective is one auditable source from which Settings model choices, runtime validation, provider defaults, capability filters, model labels and fallback selection can derive consistently.
Provider APIs may still supply dynamic availability information, but WorldScript should retain an explicit curated admission layer rather than automatically trusting every remotely advertised model.
5. Remove stale duplicated defaults
After establishing authority, reconcile every fallback/default.
Patterns such as:
must not independently drift from Settings,
AiModel, inference gateways or model selectors.The same applies to OpenAI defaults, Claude defaults, Grok defaults, OpenRouter fallback models, image-generation defaults, per-project preset defaults and local inference defaults.
Prefer shared provider-specific recommended defaults sourced from the canonical catalogue.
6. Capability-aware routing
Do not assume that every model from the same provider supports every WorldScript operation.
Build or verify capability admission for operations including:
Only include capabilities WorldScript actually uses.
If a feature requires structured output, do not route it to a model merely because that model can generate ordinary prose.
If image generation is Gemini-specific today, make that truth explicit instead of giving other provider selections the appearance that image generation will work universally.
7. Provider adapter requalification
Exercise each currently supported provider adapter against current APIs/SDKs.
Gemini
Verify current text model, current image-generation model, structured JSON, streaming where used, current SDK semantics, reasoning/thinking configuration where applicable and deprecation removal.
OpenAI
Verify current supported model IDs, reasoning model compatibility, parameter differences between model families, streaming, structured output and custom/OpenAI-compatible endpoint behavior.
Do not assume every new OpenAI model accepts exactly the same temperature/token/reasoning parameters.
Anthropic
Verify current native Claude support, exact model IDs, request structure, output extraction, cancellation, token limits and provider-specific options.
xAI / Grok
Verify current exact IDs and OpenAI-compatible/native behavior actually used by WorldScript.
OpenRouter
Verify current endpoint contract, model IDs, free/paid state, account-credit UX, fallback behavior, routing/error metadata and rate-limit handling.
A historical
:freeidentifier must not be presented as guaranteed free without current evidence.Local/OpenAI-compatible
Verify Ollama, LM Studio, vLLM and custom endpoints separately where WorldScript claims support.
Do not equate "OpenAI-compatible" with complete behavioral parity automatically.
8. Local browser AI model requalification
Audit:
For each bundled/advertised local model determine whether the model still exists, the artifact still downloads, the current library supports it, memory footprint, browser compatibility, WebGPU/WASM requirements, approximate download size, practical minimum memory, context limits, and whether it actually performs the intended writing task acceptably.
Remove or clearly classify models that are technically loadable but no longer useful as realistic user-facing defaults.
Do not replace lightweight local defaults with models that make the application unusable on ordinary hardware solely because they are newer.
9. Persisted-model migration and compatibility
Existing installations may contain historical model IDs.
Do not blindly rewrite valid user preferences.
Classify persisted states:
Required behavior:
Document all automatic migration rules.
10. Full inventory of AI product features
The second half of this issue is equally important:
First generate an authoritative inventory from all
generateText,generateJson,generateImage, streaming/inference-gateway and equivalent call sites.Known examples already include:
ai.invokecapability where enabled;Do not assume this list is complete. Derive the final inventory directly from the current repository.
11. Build an AI feature × provider × capability matrix
Produce a matrix such as:
Every cell must be classified as one of:
Avoid pretending that every provider supports every feature.
12. Real functional smoke qualification
Unit mocks are necessary but insufficient for this work.
Establish a manual/opt-in real-provider qualification suite that can exercise actual APIs without putting provider secrets into the normal repository or mandatory public CI.
Do not add production API keys to GitHub.
For each supported provider with available maintainer credentials, verify representative real requests.
At minimum test:
Text
Structured output
Images
Where supported:
Local inference
13. Test quality without brittle LLM assertions
Do not write tests expecting exact creative prose from live models.
Live qualification should assert stable properties such as request accepted, response non-empty, schema valid, language broadly correct, required keys present, image payload valid, no provider error, cancellation works and latency bounded enough for smoke qualification.
Mocked deterministic CI contract tests should cover exact adapter behavior separately.
14. Connection-test correctness
Audit every "Test connection" UI.
A green connection test must prove the configured provider/model can actually perform the minimum operation WorldScript will ask from it.
Do not report success merely because an API endpoint responds, a key has valid syntax or a provider account exists.
Differentiate errors such as:
Expose actionable UI without leaking sensitive provider responses.
15. Fallback-chain qualification
Audit hybrid fallback semantics.
Verify primary provider first, configured order respected, no duplicate provider attempts, unsupported capabilities skipped intentionally, auth failure does not create dangerous/unexpected cross-provider behavior, rate limits/transient errors follow defined policy, local-only/privacy modes cannot silently fall back to cloud and cloud fallback never bypasses explicit user privacy policy.
A fallback must not silently change from an image-capable model to a text-only provider and then surface an opaque failure.
16. AI mode qualification
Exercise all modes:
For each mode verify routing authority, model selection, settings UI, cold-start restoration, provider fallback, privacy policy, local/cloud boundaries and error messaging.
No mode should exist only as UI/state while behaving identically to another mode in runtime unless explicitly documented.
17. BYOK / secret safety
Preserve current provider-key security architecture.
During this work:
Real qualification credentials must remain external to the repository.
18. Prompt / model compatibility
Model upgrades can change behavior even when the transport works.
Run representative prompt qualification for major WorldScript use cases.
Inspect system/user role assumptions, JSON-format instructions, prompt length, thinking/reasoning controls, stop sequences, temperature/top-p support, max-token semantics, multilingual prompts and creative-writing behavior.
Do not automatically carry provider-specific parameters to a model family that rejects or ignores them.
19. Defaults by workload, not one global model everywhere
Evaluate whether one default model is appropriate for all operations.
Potential roles include:
If different workloads legitimately need different models, make that mapping explicit rather than scattering hidden exceptions.
Do not add complexity unless qualification evidence justifies it.
20. UI model catalogue correctness
Settings must show truthful current model information.
Verify provider grouping, display names, default selection, stable vs preview badges where useful, local/cloud indicator, free vs paid claims, unavailable/deprecated state, selected legacy value, capability mismatch and model access errors.
A removed model should not remain presented as a normal recommended choice.
21. Documentation and localization
After implementation, reconcile README, Help, AI/provider documentation, settings copy, provider counts, examples, model names, screenshots if model names are visible and all locale bundles through the normal i18n pipeline.
There is already evidence that provider-count/model-copy can drift between locales. Treat generated locale bundles according to repository authority rather than manually editing derived output.
Avoid prose such as "latest model" unless there is a maintainable definition or source.
22. Prevent the catalogue becoming stale again
Establish a bounded maintenance policy.
Prefer:
Consider a scheduled/advisory model-health check for providers exposing stable machine-readable model metadata.
It must not automatically rewrite production defaults, make external provider availability a flaky mandatory PR gate, or automatically trust every remotely advertised model.
It may detect deleted/deprecated configured model IDs where reliably possible and produce actionable maintenance evidence.
23. Deprecation policy
Define what happens when a provider announces model retirement.
Suggested lifecycle:
Persisted projects/settings must not become impossible to load because an AI model disappeared.
Model selection is configuration, not project-data validity.
24. Performance / cost qualification
For recommended cloud defaults capture representative latency, output quality, token usage, context capability, rate limits and approximate cost class.
This does not require a full AI benchmark laboratory. It does require enough evidence to avoid selecting a dramatically slower or substantially more expensive default merely because it is newer.
For local models capture download size, startup/load time, RAM/VRAM requirements, representative generation speed and browser/runtime support.
25. Privacy-safe qualification evidence
Record a reusable AI qualification report without storing user manuscripts or secret prompts.
For each run record only what is necessary, for example:
Do not log API keys, manuscript prose, private prompts/results, account identifiers, provider tokens or private filesystem paths.
Synthetic qualification fixtures should be used where possible.
26. CI versus real-provider testing
Maintain a clear separation.
Mandatory CI
Use deterministic mocks/fakes/contract fixtures for request construction, response parsing, model routing, fallback behavior, capability admission, persistence, cancellation and error classification.
Real provider qualification
Use controlled manual/opt-in runs for actual API availability, current model ID validity, provider behavior, real streaming, real structured output and real image generation.
Do not make ordinary PR CI depend on billable external AI APIs.
27. Architecture boundary
This issue should improve the current React/PWA/Tauri AI product without creating Tauri-only AI architecture.
Keep reusable semantics renderer-neutral where practical:
The future Qt client should be able to consume the same provider/model authority through the intended shared boundaries rather than reimplementing another catalogue.
Do not start Qt implementation from this issue.
28. Implementation slicing
This issue is broad enough that implementation will likely require multiple causal PRs.
Prefer slices such as:
Re-derive the exact split after inspection.
Do not submit one huge mixed PR if the work naturally separates.
Each PR must remain individually reviewable and reversible.
Acceptance criteria
inferenceGateway, provider adapters and UI no longer expose contradictory current/default catalogues.Non-goals
Priority / disposition
P1
AI is a primary advertised WorldScript capability. The repository already contains demonstrable model-authority drift, and the complete AI surface has not yet received systematic real-provider functional qualification.
This is therefore more than routine model-version housekeeping: stale or retired model IDs, unsupported parameter combinations or unqualified provider paths can make core advertised functionality fail at runtime despite ordinary mocked CI remaining green.