Skip to content

ai: requalify current model catalog, defaults and end-to-end AI functionality across all providers #704

Description

@qnbs

Origin / user request

WorldScript Studio's AI subsystem needs a comprehensive current-state requalification.

Two related problems need to be addressed together:

  1. Model catalogues, defaults, fallbacks and provider-specific hard-coded model IDs have accumulated over time and are no longer guaranteed to represent the best currently available supported models.
  2. The application's AI functionality has grown substantially, but the complete set of user-facing AI workflows has not yet been systematically exercised end-to-end against the real provider/runtime paths.

This issue owns a full AI model/provider refresh plus functional qualification of the complete AI product surface.

The goal is not merely to replace a few model strings. The goal is to establish:

CURRENT PROVIDER TRUTH
        ↓
CANONICAL MODEL/CAPABILITY AUTHORITY
        ↓
VETTED DEFAULTS
        ↓
CORRECT PROVIDER ADAPTERS
        ↓
REAL FUNCTIONAL QUALIFICATION
        ↓
MAINTAINABLE UPDATE / DEPRECATION POLICY

Why this is needed now

Current repository evidence already demonstrates model-authority drift.

Examples include:

  • AiModel / settings contain comparatively recent Gemini/OpenAI/Anthropic/Grok model families;
  • the canonical Gemini text fallback is still independently hard-coded in more than one runtime path;
  • DefaultInferenceGateway.modelList() still exposes an older gemini-2.0-flash entry;
  • OpenRouter has its own preferred-model/default/fallback authority;
  • model choices appear in Settings/UI, storage normalization, inference routing, provider adapters, tests, documentation and help copy;
  • local inference maintains additional independent WebLLM/ONNX/Transformers model catalogues.

This is concrete evidence that repository comments such as "current catalog" or "latest generation" cannot themselves be treated as current upstream truth.

All provider/model claims must therefore be re-derived from authoritative upstream sources at implementation time.

Do not assume that model names presently labelled "current" in source code are actually current.


1. Live upstream provider inventory

Before changing code, perform a live current-state investigation of every supported AI backend.

WorldScript currently models these provider classes:

gemini
openai
anthropic
grok
ollama
openrouter
webllm
onnx
transformers

Also account for local OpenAI-compatible runtime presets where applicable, including:

Ollama
LM Studio
vLLM
custom OpenAI-compatible endpoint

For each externally managed provider, inspect the official provider documentation/API at execution time.

Record at minimum:

Field Required
exact model ID yes
human display name yes
availability state stable / preview / experimental / deprecated / shutdown
text generation yes/no
streaming yes/no
structured/JSON output yes/no
image input yes/no
image generation yes/no
tool/function calling yes/no where relevant
reasoning support yes/no/variant
context limit where authoritative
output-token limit where relevant
provider-specific parameter restrictions yes
API/SDK compatibility yes
pricing/free-tier status when surfaced in WorldScript UX
deprecation/shutdown date where published
recommended replacement where published

Prefer official documentation, model-list APIs and official deprecation notices over blog posts, third-party model lists or repository comments.

For OpenRouter, verify current OpenRouter metadata rather than assuming a historical :free model remains free or available.


2. Separate "newest" from "best default"

Do not mechanically make every newest preview/experimental model the application default.

Establish explicit model lifecycle categories such as:

RECOMMENDED_DEFAULT
CURRENT_SUPPORTED
SPECIALIZED
PREVIEW_OPT_IN
LEGACY_COMPAT
DEPRECATED
REMOVED_UPSTREAM
UNKNOWN_STORED_VALUE

Selection of the WorldScript default should consider:

quality
stability
latency
availability
cost
rate limits
context window
structured-output reliability
streaming support
provider compatibility
creative-writing quality

A newer experimental model may be listed as an opt-in choice without replacing a proven stable production default.

Conversely, a deprecated/shutdown model must not remain an active default merely because it still exists in persisted types.


3. Inventory every model authority in the repository

Perform a repository-wide model-string and provider-authority audit.

At minimum inspect:

  • types.ts / AiModel / AIProvider;
  • features/settings/settingsSlice.ts;
  • components/settings/AiSections.tsx;
  • components/settings/OpenRouterSection.tsx;
  • services/geminiService.ts;
  • services/aiProviderService.ts;
  • services/ai/providerFactory.ts;
  • services/ai/inferenceGateway.ts;
  • services/ai/worldScriptCompletionFetch.ts;
  • services/ai/aiModeService.ts;
  • services/ai/aiConstants.ts;
  • hybrid fallback/routing;
  • provider connection testing;
  • IDB/settings normalization;
  • filesystem/cloud settings persistence;
  • per-project AI overrides;
  • ProForge AI configuration;
  • local AI model registries;
  • WebLLM supported models;
  • ONNX supported models;
  • Transformers.js model assumptions;
  • image-generation model IDs;
  • documentation/help/locales;
  • tests and fixtures;
  • historical compatibility paths.

Search for all literal model IDs, not only known defaults.

Classify each occurrence as:

CANONICAL_AUTHORITY
DERIVED_VIEW
PERSISTED_LEGACY_COMPAT
TEST_FIXTURE
DOCUMENTATION
STALE_DUPLICATE_AUTHORITY
REMOVE

A model ID should not silently acquire multiple independent "default" authorities.


4. Establish a canonical model registry

Where practical, replace scattered current-model knowledge with one explicit capability-aware authority.

A model entry should be able to represent properties conceptually equivalent to:

provider
modelId
displayName
lifecycle
capabilities
stability
recommendedRoles
requiresApiKey
localOrCloud
supportsStreaming
supportsStructuredOutput
supportsImageInput
supportsImageGeneration
supportsReasoning
contextLimit
outputLimit
deprecation
replacementModel

Do not over-engineer this into a remote model-management platform.

The immediate objective is one auditable source from which Settings model choices, runtime validation, provider defaults, capability filters, model labels and fallback selection can derive consistently.

Provider APIs may still supply dynamic availability information, but WorldScript should retain an explicit curated admission layer rather than automatically trusting every remotely advertised model.


5. Remove stale duplicated defaults

After establishing authority, reconcile every fallback/default.

Patterns such as:

model?.startsWith('gemini-') ? model : '<hard-coded-model>'

must not independently drift from Settings, AiModel, inference gateways or model selectors.

The same applies to OpenAI defaults, Claude defaults, Grok defaults, OpenRouter fallback models, image-generation defaults, per-project preset defaults and local inference defaults.

Prefer shared provider-specific recommended defaults sourced from the canonical catalogue.


6. Capability-aware routing

Do not assume that every model from the same provider supports every WorldScript operation.

Build or verify capability admission for operations including:

TEXT
STREAMING_TEXT
STRUCTURED_JSON
IMAGE_INPUT
IMAGE_GENERATION
REASONING
TOOLS / FUNCTION CALLING
EMBEDDINGS / RAG

Only include capabilities WorldScript actually uses.

If a feature requires structured output, do not route it to a model merely because that model can generate ordinary prose.

If image generation is Gemini-specific today, make that truth explicit instead of giving other provider selections the appearance that image generation will work universally.


7. Provider adapter requalification

Exercise each currently supported provider adapter against current APIs/SDKs.

Gemini

Verify current text model, current image-generation model, structured JSON, streaming where used, current SDK semantics, reasoning/thinking configuration where applicable and deprecation removal.

OpenAI

Verify current supported model IDs, reasoning model compatibility, parameter differences between model families, streaming, structured output and custom/OpenAI-compatible endpoint behavior.

Do not assume every new OpenAI model accepts exactly the same temperature/token/reasoning parameters.

Anthropic

Verify current native Claude support, exact model IDs, request structure, output extraction, cancellation, token limits and provider-specific options.

xAI / Grok

Verify current exact IDs and OpenAI-compatible/native behavior actually used by WorldScript.

OpenRouter

Verify current endpoint contract, model IDs, free/paid state, account-credit UX, fallback behavior, routing/error metadata and rate-limit handling.

A historical :free identifier must not be presented as guaranteed free without current evidence.

Local/OpenAI-compatible

Verify Ollama, LM Studio, vLLM and custom endpoints separately where WorldScript claims support.

Do not equate "OpenAI-compatible" with complete behavioral parity automatically.


8. Local browser AI model requalification

Audit:

WebLLM
ONNX Runtime Web
Transformers.js

For each bundled/advertised local model determine whether the model still exists, the artifact still downloads, the current library supports it, memory footprint, browser compatibility, WebGPU/WASM requirements, approximate download size, practical minimum memory, context limits, and whether it actually performs the intended writing task acceptably.

Remove or clearly classify models that are technically loadable but no longer useful as realistic user-facing defaults.

Do not replace lightweight local defaults with models that make the application unusable on ordinary hardware solely because they are newer.


9. Persisted-model migration and compatibility

Existing installations may contain historical model IDs.

Do not blindly rewrite valid user preferences.

Classify persisted states:

CURRENT_SUPPORTED
DEPRECATED_BUT_AVAILABLE
REMOVED_UPSTREAM
UNKNOWN
MALFORMED

Required behavior:

  • current supported explicit selection → preserve;
  • deprecated but still usable → preserve or warn according to policy;
  • removed/shutdown → offer/use a defined safe replacement;
  • unknown future/custom model → preserve where the provider supports arbitrary IDs rather than destructively normalizing it away;
  • invalid combinations → fail with actionable UX rather than silently routing to an unrelated provider.

Document all automatic migration rules.


10. Full inventory of AI product features

The second half of this issue is equally important:

Every actual AI feature exposed by WorldScript must be exercised and qualified.

First generate an authoritative inventory from all generateText, generateJson, generateImage, streaming/inference-gateway and equivalent call sites.

Known examples already include:

  • Writer/completion workflows;
  • manuscript continuation/generation;
  • synopsis generation;
  • character generation and field regeneration;
  • world generation and field regeneration;
  • Critic analysis;
  • Consistency Checker;
  • outline/story planning flows;
  • scene-related generation;
  • character interviews where AI-backed;
  • scene visualization;
  • character portraits;
  • world images;
  • ProForge pipeline agents;
  • AI API/internal orchestration;
  • plugin ai.invoke capability where enabled;
  • model/provider connection tests;
  • RAG/local-routing flows;
  • per-project AI presets;
  • hybrid fallback paths.

Do not assume this list is complete. Derive the final inventory directly from the current repository.


11. Build an AI feature × provider × capability matrix

Produce a matrix such as:

Feature Text JSON Image Streaming Gemini OpenAI Claude Grok OpenRouter Local
Continue writing
Critic
Consistency structured/parsing
Character generation ✓/text
Character portrait

Every cell must be classified as one of:

SUPPORTED_AND_QUALIFIED
SUPPORTED_NOT_YET_QUALIFIED
UNSUPPORTED_BY_PROVIDER
BLOCKED_BY_POLICY
LOCAL_ONLY
CLOUD_ONLY
NOT_APPLICABLE

Avoid pretending that every provider supports every feature.


12. Real functional smoke qualification

Unit mocks are necessary but insufficient for this work.

Establish a manual/opt-in real-provider qualification suite that can exercise actual APIs without putting provider secrets into the normal repository or mandatory public CI.

Do not add production API keys to GitHub.

For each supported provider with available maintainer credentials, verify representative real requests.

At minimum test:

Text

  • simple generation;
  • long-form writing continuation;
  • cancellation;
  • streaming where supported;
  • non-empty result;
  • Unicode/non-English text.

Structured output

  • valid expected schema;
  • malformed-model-output recovery;
  • parser behavior;
  • retry/fallback behavior.

Images

Where supported:

  • character portrait;
  • world image;
  • scene visualization;
  • valid returned artifact;
  • error/cancellation path.

Local inference

  • model initialization/download;
  • successful inference;
  • offline behavior after model availability;
  • unsupported browser/runtime behavior;
  • cancellation;
  • memory/resource cleanup.

13. Test quality without brittle LLM assertions

Do not write tests expecting exact creative prose from live models.

Live qualification should assert stable properties such as request accepted, response non-empty, schema valid, language broadly correct, required keys present, image payload valid, no provider error, cancellation works and latency bounded enough for smoke qualification.

Mocked deterministic CI contract tests should cover exact adapter behavior separately.


14. Connection-test correctness

Audit every "Test connection" UI.

A green connection test must prove the configured provider/model can actually perform the minimum operation WorldScript will ask from it.

Do not report success merely because an API endpoint responds, a key has valid syntax or a provider account exists.

Differentiate errors such as:

NO_API_KEY
INVALID_KEY
MODEL_NOT_FOUND
MODEL_DEPRECATED
MODEL_ACCESS_DENIED
INSUFFICIENT_CREDITS
RATE_LIMITED
NETWORK_UNAVAILABLE
LOCAL_BACKEND_OFFLINE
CAPABILITY_UNSUPPORTED
PROVIDER_ERROR

Expose actionable UI without leaking sensitive provider responses.


15. Fallback-chain qualification

Audit hybrid fallback semantics.

Verify primary provider first, configured order respected, no duplicate provider attempts, unsupported capabilities skipped intentionally, auth failure does not create dangerous/unexpected cross-provider behavior, rate limits/transient errors follow defined policy, local-only/privacy modes cannot silently fall back to cloud and cloud fallback never bypasses explicit user privacy policy.

A fallback must not silently change from an image-capable model to a text-only provider and then surface an opaque failure.


16. AI mode qualification

Exercise all modes:

HYBRID
CLOUD
LOCAL
ECO

For each mode verify routing authority, model selection, settings UI, cold-start restoration, provider fallback, privacy policy, local/cloud boundaries and error messaging.

No mode should exist only as UI/state while behaving identically to another mode in runtime unless explicitly documented.


17. BYOK / secret safety

Preserve current provider-key security architecture.

During this work:

  • never commit API keys;
  • never include keys in logs;
  • never persist provider keys in general Redux/project serialization;
  • retain dedicated encrypted key-store authority;
  • never expose secrets to plugins;
  • redact provider errors where they may contain account/request metadata;
  • do not weaken CSP/storage/privacy policy merely to support a provider.

Real qualification credentials must remain external to the repository.


18. Prompt / model compatibility

Model upgrades can change behavior even when the transport works.

Run representative prompt qualification for major WorldScript use cases.

Inspect system/user role assumptions, JSON-format instructions, prompt length, thinking/reasoning controls, stop sequences, temperature/top-p support, max-token semantics, multilingual prompts and creative-writing behavior.

Do not automatically carry provider-specific parameters to a model family that rejects or ignores them.


19. Defaults by workload, not one global model everywhere

Evaluate whether one default model is appropriate for all operations.

Potential roles include:

GENERAL_WRITING_DEFAULT
FAST_LOW_COST_DEFAULT
HIGH_QUALITY_REASONING
STRUCTURED_OUTPUT
IMAGE_GENERATION
LOCAL_LIGHTWEIGHT
OPENROUTER_FREE

If different workloads legitimately need different models, make that mapping explicit rather than scattering hidden exceptions.

Do not add complexity unless qualification evidence justifies it.


20. UI model catalogue correctness

Settings must show truthful current model information.

Verify provider grouping, display names, default selection, stable vs preview badges where useful, local/cloud indicator, free vs paid claims, unavailable/deprecated state, selected legacy value, capability mismatch and model access errors.

A removed model should not remain presented as a normal recommended choice.


21. Documentation and localization

After implementation, reconcile README, Help, AI/provider documentation, settings copy, provider counts, examples, model names, screenshots if model names are visible and all locale bundles through the normal i18n pipeline.

There is already evidence that provider-count/model-copy can drift between locales. Treat generated locale bundles according to repository authority rather than manually editing derived output.

Avoid prose such as "latest model" unless there is a maintainable definition or source.


22. Prevent the catalogue becoming stale again

Establish a bounded maintenance policy.

Prefer:

official upstream verification
        ↓
curated registry update
        ↓
focused provider contract tests
        ↓
real qualification
        ↓
release

Consider a scheduled/advisory model-health check for providers exposing stable machine-readable model metadata.

It must not automatically rewrite production defaults, make external provider availability a flaky mandatory PR gate, or automatically trust every remotely advertised model.

It may detect deleted/deprecated configured model IDs where reliably possible and produce actionable maintenance evidence.


23. Deprecation policy

Define what happens when a provider announces model retirement.

Suggested lifecycle:

CURRENT
  ↓
DEPRECATION ANNOUNCED
  ↓
REPLACEMENT QUALIFIED
  ↓
DEFAULT MOVED
  ↓
LEGACY SELECTION WARNED
  ↓
UPSTREAM SHUTDOWN
  ↓
REMOVED FROM NORMAL PICKER

Persisted projects/settings must not become impossible to load because an AI model disappeared.

Model selection is configuration, not project-data validity.


24. Performance / cost qualification

For recommended cloud defaults capture representative latency, output quality, token usage, context capability, rate limits and approximate cost class.

This does not require a full AI benchmark laboratory. It does require enough evidence to avoid selecting a dramatically slower or substantially more expensive default merely because it is newer.

For local models capture download size, startup/load time, RAM/VRAM requirements, representative generation speed and browser/runtime support.


25. Privacy-safe qualification evidence

Record a reusable AI qualification report without storing user manuscripts or secret prompts.

For each run record only what is necessary, for example:

provider
model
feature
result
capability
duration
error category
environment
date
application SHA

Do not log API keys, manuscript prose, private prompts/results, account identifiers, provider tokens or private filesystem paths.

Synthetic qualification fixtures should be used where possible.


26. CI versus real-provider testing

Maintain a clear separation.

Mandatory CI

Use deterministic mocks/fakes/contract fixtures for request construction, response parsing, model routing, fallback behavior, capability admission, persistence, cancellation and error classification.

Real provider qualification

Use controlled manual/opt-in runs for actual API availability, current model ID validity, provider behavior, real streaming, real structured output and real image generation.

Do not make ordinary PR CI depend on billable external AI APIs.


27. Architecture boundary

This issue should improve the current React/PWA/Tauri AI product without creating Tauri-only AI architecture.

Keep reusable semantics renderer-neutral where practical:

provider identity
model identity
capabilities
routing
policy
request/response contracts

The future Qt client should be able to consume the same provider/model authority through the intended shared boundaries rather than reimplementing another catalogue.

Do not start Qt implementation from this issue.


28. Implementation slicing

This issue is broad enough that implementation will likely require multiple causal PRs.

Prefer slices such as:

A. live inventory + canonical model authority
B. provider/default/catalogue reconciliation
C. cloud provider contract corrections
D. local model requalification
E. application-wide AI functional qualification
F. docs/UX/maintenance policy

Re-derive the exact split after inspection.

Do not submit one huge mixed PR if the work naturally separates.

Each PR must remain individually reviewable and reversible.


Acceptance criteria

  • All nine declared AI providers are inventoried and classified against current implementation reality.
  • Current upstream model availability is re-fetched from authoritative provider sources at implementation time.
  • Every literal production model ID has a documented authority/classification.
  • No deprecated/shutdown model remains an accidental active default.
  • Recommended defaults are current, supported and qualification-backed.
  • "Newest" and "recommended default" are explicitly distinguished.
  • Model lifecycle states are represented explicitly.
  • Settings, runtime routing, fallback logic and model lists derive from consistent authority.
  • inferenceGateway, provider adapters and UI no longer expose contradictory current/default catalogues.
  • Stored historical model selections have defined preserve/migrate/failure semantics.
  • OpenRouter free/paid claims are verified against current truth.
  • Local WebLLM/ONNX/Transformers models are requalified for availability and practical usability.
  • Ollama/LM Studio/vLLM/custom local endpoints are exercised according to claimed support.
  • Every user-visible AI feature has been inventoried.
  • A feature × provider × capability support matrix exists.
  • Representative text generation works against real qualified providers.
  • Structured-output paths are qualified.
  • Streaming paths are qualified where supported.
  • Character/world/scene image-generation paths are qualified against the actual image model.
  • Critic and Consistency Checker are functionally qualified.
  • Writer/manuscript generation workflows are functionally qualified.
  • Character/world generation workflows are functionally qualified.
  • ProForge/provider-agent paths are reconciled where currently supported.
  • AI connection-test UX proves meaningful provider/model usability.
  • Cancellation, rate-limit, unavailable-model and auth-error paths are tested.
  • Hybrid/cloud/local/eco routing is verified.
  • Local-only/privacy policy cannot silently fall back to cloud.
  • No real credentials are committed or exposed in logs.
  • Real-provider smoke qualification is separated from deterministic mandatory CI.
  • AI documentation/settings/help copy matches actual current behavior.
  • A bounded ongoing model/deprecation maintenance policy prevents the catalogue from becoming silently obsolete again.
  • All implementation PRs pass normal exact-head CI/CodeQL/review convergence and resulting-main verification.

Non-goals

  • automatically adopting every new preview/experimental model;
  • claiming the newest model is necessarily the best WorldScript default;
  • putting paid external AI calls into mandatory PR CI;
  • committing maintainer/provider credentials;
  • deleting old persisted model values without compatibility handling;
  • automatically trusting arbitrary models returned by a remote catalogue;
  • rewriting the entire AI architecture unnecessarily;
  • making every provider support every AI feature artificially;
  • weakening privacy/local-only policy for fallback convenience;
  • starting Qt implementation;
  • benchmarking every model on the market;
  • turning WorldScript into a generic provider/model marketplace.

Priority / disposition

P1

AI is a primary advertised WorldScript capability. The repository already contains demonstrable model-authority drift, and the complete AI surface has not yet received systematic real-provider functional qualification.

This is therefore more than routine model-version housekeeping: stale or retired model IDs, unsupported parameter combinations or unqualified provider paths can make core advertised functionality fail at runtime despite ordinary mocked CI remaining green.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions