test(ai-e2e): qualify Copilot, Character Interviews, ProForge and mutation-capable AI user flows

[qnbs]

Context

Child/follow-up of #704 and dependent on the coverage-authority foundation in #709.

Current WorldScript AI E2E evidence is uneven:

• Outline generation already has a real deterministic browser flow using the real application stack while mocking only the external Gemini HTTP boundary.
• Global Copilot has a dedicated flag/UI scaffold E2E, but that spec explicitly makes no AI calls.
• ProForge has exposure/scaffold coverage but not a representative bounded pipeline transition through real orchestration.
• Character Interviews are AI-backed, streaming and mutation-capable, yet current E2E search did not find a dedicated functional interview flow.
• Current E2E search likewise found no clear product-flow coverage for several mutation-capable AI surfaces such as portrait generation, proofread, AI synopsis and scene visualization.

This issue owns deterministic, secret-free, product-level functional E2E qualification of high-risk AI surfaces.

It does not own provider/model freshness itself (#704), cross-project mutation source fixes (#708 and Wave 0B follow-ups), or broad feature-flag coverage governance (#709).

---

Core testing principle

For mandatory deterministic AI E2E, use:

REAL BROWSER UI
        ↓
REAL HOOK / THUNK / SERVICE
        ↓
REAL REDUX + PROJECT STATE
        ↓
REAL PROVIDER ROUTING / ADAPTER
        ↓
MOCK EXTERNAL PROVIDER TRANSPORT ONLY
        ↓
DETERMINISTIC RESPONSE
        ↓
ASSERT USER-VISIBLE RESULT / MUTATION CONTRACT

Use the existing Outline/Gemini interception pattern as the architectural reference where appropriate.

Do not make required PR CI depend on:

• live cloud provider uptime;
• paid inference;
• maintainer secrets;
• exact creative prose from real models.

Real-provider qualification remains a separate manual/nightly/opt-in layer under #704.

---

1. Global Copilot — current scaffold coverage is insufficient

Current tests/e2e/copilot-flags.spec.ts proves UI/flag behavior such as launcher/dialog visibility and disabled state, but intentionally performs no AI request.

Therefore it does not currently prove:

send
context construction
provider routing
streaming/response
assistant rendering
apply-to-manuscript
project authority
Stop Generation/error recovery

Required deterministic functional flow

Add at least one required product E2E:

create/open deterministic project
explicitly enable Global Copilot
seed minimum deterministic provider configuration
open Copilot
send user message
intercept external provider request
return deterministic response
assert assistant result is rendered

Where stable, inspect a bounded part of the outgoing request/context so the test proves the selected project's context is actually entering the request path without snapshotting large prompt blobs.

Copilot apply contract

Add a representative mutation-capable flow:

project A active
Copilot returns a known applicable suggestion/code block
user invokes Apply
expected manuscript section in A changes exactly as intended

After the relevant Wave 0B project-authority source fixes are terminal, add the regression:

result originates under A
switch/reset/import/restore to B
attempt Apply
→ B remains unchanged

Use same section IDs across A/B where practical so the test cannot pass merely because IDs differ.

Do not implement the source authority fix in this test issue if #708/successor already owns it; land tests after the source contract exists.

---

2. Character Interviews — functional streaming qualification

Character Interviews are especially important because they combine:

• AI context derived from character/project state;
• streaming callbacks;
• transcript mutation;
• project-switch authority;
• cancellation lifecycle.

Add a deterministic functional browser flow:

enable Character Interviews explicitly
create/select deterministic character
open Character Interviews
create/start interview
submit user message
intercept provider transport
emit deterministic response/chunks
assert transcript updates correctly

Project-authority regression

After #708 or its successor fixes the source path:

start stream under project A
allow first chunk
switch/reset/import/restore to B
emit late chunk(s)
→ B interview state remains unchanged

Prefer same characterId / interviewId fixtures between A and B where feasible.

Cancellation regression

General Stop Generation lifecycle belongs to #704 Wave 0C rather than #708.

Once that contract lands, add:

stream starts
user presses Stop
late transport chunks/results still arrive
→ no later transcript mutation
→ UI settles to the defined cancelled state

Do not pre-design cancellation semantics inside this issue; test the admitted product contract once source ownership is clear.

---

3. ProForge — prove one bounded real orchestration transition

Current ProForge exposure tests are not equivalent to proving the agentic pipeline wiring.

Do not run a full paid eight-stage production pipeline in mandatory CI.

Add the smallest deterministic functional product flow that exercises real orchestration, e.g.:

enable ProForge
open Writer/ProForge
select/start one bounded admitted stage/action
intercept provider/service transport at the external boundary
return deterministic result
assert expected ProForge state transition / output / HITL gate

Where Human-in-the-Loop is core to the path, qualify one representative approval or reject/apply transition.

Do not mock the pipeline/state-machine layer that the test is intended to prove.

---

4. Build an authoritative AI product-surface E2E inventory

Before adding many tests, derive the current product surface from live source.

Search all production AI call sites including generateText, generateJson, generateImage, streaming/inference-gateway calls and equivalent wrappers.

Build a matrix:

AI surface	User entry	Flag	Default	Text/JSON/Image	Streaming	Mutation/apply	Current functional E2E	Required disposition

Candidate surfaces to verify include, without assuming completeness:

Writer generation/continuation
rewrite/improve/change-tone/expand/shorten
brainstorming
outline generation
outline section regeneration
AI template personalization/custom generation
character generation
character field regeneration
character portrait/refinement
Character Interviews
world generation
world field regeneration
world image/refinement
logline suggestions
proofread
AI synopsis
scene visualization
Critic/review
Consistency Checker
Global Copilot
ProForge
RAG/context-assisted generation
command-palette AI actions
background AI work
plugin ai.invoke where user-facing/admitted

Reconcile this matrix with #709's coverage registry so there is one explicit disposition rather than a disconnected spreadsheet/document.

---

5. Prioritize mutation-capable user flows

Browser functional E2E should prioritize AI operations that can change user work or persistent assets.

Examples:

Apply outline
Apply personalized template
Writer Accept insert/replace
Copilot Apply
character/world profile mutations
field regeneration
Character Interview transcript mutation
logline selection
proofread apply
AI synopsis entering export
character/world/scene image creation
ProForge approval/apply

Detailed races remain best covered at unit/hook/thunk/reducer/storage level.

Browser E2E should prove representative complete user journeys, not every race permutation.

---

6. Character/world/scene image functional qualification

After project-authority and storage ownership work under #708/successors is terminal, add representative browser evidence for AI image generation.

At least prove:

user invokes image generation
real thunk/service/provider routing executes
external image provider request is deterministically mocked
valid deterministic artifact returns
UI/state reflects successful image availability

Choose bounded representative coverage across Character / World / Scene based on actual shared-vs-distinct code paths.

If all three have materially distinct thunks/storage semantics, qualify all three.

If one common path genuinely provides the same contract, avoid redundant browser tests and keep surface-specific races at lower layers.

Do not use real image-generation APIs in required CI.

---

7. Proofread, logline, synopsis and deferred-result application

These paths are important because a result may be generated now and applied/exported later.

After Wave 0B deferred-result ownership fixes land, add representative functional E2E where practical for:

Proofread

generate deterministic suggestion
suggestion appears
apply suggestion
manuscript changes as intended

and, where stable:

generate under A
switch to B
stale apply unavailable/rejected

Logline

generate suggestions
select deterministic suggestion
current project logline updates

AI synopsis / export

generate deterministic synopsis
export/preview includes it for originating project
switch to B invalidates old synopsis for B export

Do not duplicate lower-level authority tests unnecessarily; one browser journey per distinct user-facing contract is sufficient.

---

8. Writer functional AI coverage

Current full-project E2E reaches Writer but mainly edits the Writer textbox manually; it is not by itself proof of AI generation/stream/apply.

Audit both production Writer paths:

useWorldScriptAI orchestration
legacy streamGenerationThunk

Add a representative deterministic generation + accept path against the real application wiring.

If both paths remain production-reachable and materially distinct, both need explicit qualification at some test layer.

Browser E2E may test one primary user path while lower-level tests prove the compatibility/legacy path, provided #709 records that disposition explicitly.

After Wave 0B ownership fixes:

A-origin Writer result
switch to B
Accept insert/replace
→ B unchanged

should be represented at least once at product level if it can be made deterministic without large harness complexity.

---

9. Error-path qualification

For each new required AI functional E2E, add only the error cases that prove distinct product behavior.

Representative categories:

provider unreachable
invalid/malformed deterministic response
rate-limit classification fixture
cancelled request where admitted
capability unsupported

Do not build every provider error × every feature at browser level.

Adapter/service unit/integration tests should carry the detailed provider matrix.

Browser E2E should prove the feature surfaces actionable UX and does not corrupt project state.

---

10. Provider mocking boundary

Prefer HTTP/transport interception such as Playwright page.route(...) against the actual provider endpoint where architecture permits.

Avoid mocking:

React component
useGlobalCopilot/useWriterView hook
Redux dispatch
project reducer
entire AI service

when the test claims to be a functional product E2E.

If a provider path cannot currently be intercepted cleanly because transport is too entangled, record that architecture evidence rather than weakening the test into a component mock and calling it E2E.

---

11. No brittle LLM-output assertions

Deterministic mocked E2E may assert exact synthetic fixture content.

Real-provider qualification under #704 should assert only stable properties such as:

request accepted
response non-empty
schema valid
required keys present
image payload valid
stream completes/cancels
error category correct

Do not require exact live creative prose.

---

12. Required versus advisory placement

The small deterministic happy-path and critical mutation/apply journeys in this issue should normally live in the required ordinary E2E lane once stable.

Broad combinations, expensive stress, or real-runtime/model work may remain in e2e-deep, nightly or manual lanes.

Coordinate any advisory-lane graduation policy with #623.

Do not make the entire deep suite blocking from this issue.

---

13. Mobile/desktop browser scope

Current required Playwright CI runs Desktop Chromium + Mobile Chrome.

A test should run on both only when the feature is product-admitted on both and the assertions make semantic sense.

If a feature is desktop-layout-only or mobile-inapplicable, encode that honestly rather than adding .catch() or weak assertions to force a pass.

Do not add Firefox/WebKit to required CI from this issue.

---

14. Relationship to active correctness work

#708 owns

Cross-project mutation/asset authority for:

• Character portrait/refinement;
• World image/refinement;
• image storage ownership;
• Character Interview streaming.

Do not duplicate those source fixes here.

#704 Wave 0B/Wave 0C owns

• broader project-result ownership;
• cancellation/duplicate-request lifecycle;
• provider/model functional qualification.

This issue supplies deterministic product evidence once those contracts are admitted.

#709 owns

The exhaustive feature-test coverage authority and required/advisory classification machinery.

#623 owns

Deep-E2E advisory graduation/expiry evidence.

---

Acceptance criteria

• A current AI product-surface × E2E coverage matrix is derived from live production call sites.
• Global Copilot has a deterministic real UI → real routing → mocked transport → rendered response E2E.
• Global Copilot has at least one representative Apply-to-project E2E once source authority is admitted.
• Character Interviews have a deterministic real streaming transcript E2E.
• Character Interview cross-project late-chunk regression is product-qualified after ai(project): bind asynchronous image and interview side effects to originating project #708 source correction.
• ProForge has at least one bounded functional pipeline/HITL transition through real orchestration.
• Writer AI generation/accept has an explicit functional qualification disposition and representative product test.
• High-risk deferred apply/export surfaces (proofread/logline/synopsis or equivalent) have representative product E2E after source ownership fixes.
• Character/world/scene image generation has representative deterministic browser qualification after storage/project authority is safe.
• Required AI E2E mocks only external provider/runtime boundaries, not the application orchestration being claimed.
• Mandatory CI uses no live paid provider credentials.
• Real-provider qualification remains separate under ai: requalify current model catalog, defaults and end-to-end AI functionality across all providers #704.
• Required vs advisory placement is explicit and reconciled with test(feature-flags): establish exhaustive coverage authority and prevent opt-in E2E blind spots #709/ci: define e2e-deep advisory graduation and expiry evidence #623.
• Tests do not mask E2E WelcomePortal entry remains nondeterministic across startup states #532-style harness failures with broad retries/timeouts/catches.
• Focused specs, pnpm run ci:prepush, exact-head CI/CD and CodeQL are green before each implementation PR merges.

---

Non-goals

• every feature × every provider browser permutation;
• paid/live cloud inference in required CI;
• exact prose assertions against live models;
• implementing ai(project): bind asynchronous image and interview side effects to originating project #708 project-authority source fixes inside test-only PRs;
• redesigning Wave 0C cancellation semantics here;
• rewriting provider architecture merely to make a test shorter;
• full cross-browser expansion;
• replacing lower-level race/adapter tests with Playwright.

---

Priority / sequencing

P1 functional evidence for #704.

Recommended order:

#708 / remaining Wave 0B data-integrity source fixes
        ↓
#709 coverage authority
        ↓
THIS ISSUE — functional AI E2E
        ↓
#623 evidence-driven promotion of additional stable deep scenarios

Individual tests may land alongside their corresponding source PR when doing so is the smallest causal change, but avoid one giant mixed AI-E2E PR.

[qnbs]

Source-owner reconciliation addendum — #713 / #714

The source-correctness ownership referenced in this test issue has now been split into durable child issues and should be treated as follows:

#708
DIRECT / PERSISTENT / STREAMING PROJECT AUTHORITY
- Character portrait/refinement
- World image/refinement
- Scene image persistent-write path
- image-storage ownership
- Character Interview stream mutation authority

#713
DEFERRED / APPLY-LATER PROJECT AUTHORITY
- Writer result/history ownership
- Writer Accept insert/replace
- Global Copilot result/apply ownership
- outline regeneration preview
- logline suggestions
- proofread suggestions
- AI synopsis/export contamination
- stale Scene/other previews

#714
REQUEST LIFECYCLE
- Stop Generation
- AbortSignal propagation
- cancellation
- supersession/latest-request
- duplicate-request admission
- streaming late-event admission
- timeout/fallback cancellation semantics

Implication for #710

#710 remains the product-level deterministic functional E2E owner. It must not absorb source fixes already owned by #708/#713/#714.

For each browser journey, land or enable the test only after the relevant source contract is admitted enough that the test is qualifying intended product behavior rather than forcing architecture from Playwright.

Recommended dependencies:

Writer stale apply E2E                → #713
Copilot stale apply E2E               → #713
Character Interview cross-project     → #708
Character/World/Scene image ownership → #708
Stop Generation / late chunks         → #714
supersession/latest-result behavior    → #714

Where one user journey composes multiple guarantees, assert the observable final invariant while retaining lower-layer tests under their source owners.

Example:

start stream under A
→ Stop
→ switch to B with same target ids
→ late chunks arrive
→ B unchanged and UI remains terminal

This exercises both lifecycle and project authority, but does not merge their implementation ownership.

Mandatory CI remains deterministic/secret-free and should continue mocking only external provider/runtime boundaries where architecture permits.

[qnbs]

Addendum — #736 governs required-gate placement and scaling for AI E2E

#710 remains the functional AI product-E2E owner. #736 now owns whether/how those scenarios enter the protected ordinary Playwright gate.

Mandatory AI E2E contract

For any scenario that is required or a candidate for promotion, keep the existing deterministic architecture strict:

REAL browser UI
→ REAL hook/thunk/service/router/provider adapter
→ mock ONLY external provider transport/runtime boundary
→ deterministic fixture response
→ assert real user-visible / mutation contract

Required PR CI must not depend on:

• OpenAI/Gemini/Anthropic/xAI/OpenRouter secrets;
• paid/billable inference;
• provider uptime;
• exact live creative prose;
• a real local Ollama daemon;
• large real local-model downloads.

Real-provider/model qualification remains separate under #704.

Default lane for new AI E2E

Do not assume every new #710 scenario belongs in required CI immediately.

Classify each scenario against #736:

REQUIRED_CRITICAL_JOURNEY
ADVISORY_AI_E2E
ADVISORY_DEEP
REAL_RUNTIME_QUALIFICATION

A mutation/privacy/data-integrity journey that is deterministic, secret-free and uniquely protects merge safety may belong in required E2E once stable. Broad provider matrices, expensive stress/error permutations and real-runtime/model paths remain advisory/manual/nightly.

Promotion evidence

Before an advisory AI scenario is considered for required status, require at least 10 clean first-attempt main runs for that exact scenario/path. A clean run means no retry, flake classification, skip/quarantine/catch-based suppression, or external-provider dependency.

This is in addition to the currently documented #623 / docs/CI.md deep-E2E graduation window of 3 consecutive weeks of main runs unless that policy is explicitly changed with evidence.

Locale-safe interaction

AI surfaces must follow #736's locator policy. WorldScript has 19 locale bundles; ordinary Copilot/Interview/Writer/ProForge navigation should not be coupled to one English string when stable data-tour, data-testid, or locale-independent semantics are available.

Do not add test IDs indiscriminately; add/reuse durable anchors at the user interaction boundaries the E2E actually needs.

Desktop/mobile placement

For every #710 scenario, explicitly decide:

DESKTOP_AND_MOBILE
DESKTOP_ONLY
MOBILE_ONLY

Do not force desktop-only ProForge or other surfaces through Mobile Chrome with weak .catch()/conditional assertions.

Scale-out

The current CI process deliberately uses workers: 1. As #710 grows, do not solve runtime by raising workers or changing frameworks first. #736/#550 own a measured Playwright-native sharding path (--shard=1/2, --shard=2/2, still workers: 1 per process) when p50/p95 runtime evidence justifies it.

Web vs native

#710 browser E2E proves React/PWA product behavior. It must not pretend to qualify native-only filesystem, updater, native-dialog, packaged-window or IPC semantics. Those belong in separate native qualification under #736/#332 and specialist native owners.

Additional acceptance implications for #710:

• every new AI E2E has an explicit required/advisory/real-runtime disposition;
• required AI E2E is secret-free and transport-mocked only at the external boundary;
• promotion uses scenario-level stability evidence, not aggregate e2e-deep greenness;
• translated visible strings are not the default primary interaction locator;
• Desktop/Mobile applicability is explicit;
• suite growth remains shard-ready and order-independent;
• browser AI E2E does not duplicate native-only qualification.

[qnbs]

Current source/evidence dependency checkpoint — 2026-09-16

This issue remains the deterministic product-level AI E2E owner, but its body still reflects the earlier #704/#708 ownership topology. Use the following current graph when implementing new browser journeys:

Source correctness predecessors

• ai(project): bind asynchronous image and interview side effects to originating project #708 is terminal/closed.
• fix(ai): project-qualify image storage and use the real delete API #719 is terminal for project-qualified image/storage ownership.
• fix(project): fence async image authority by incarnation #734 is terminal for direct Character/World/Scene image authority and Character Interview project-incarnation fences.
• fix(ai): reject stale AI image writes and project-qualify image storage #718 was reconciled and closed as superseded; do not build E2E expectations around that stale branch.
• ai(project): bind Writer, Copilot and deferred AI results to originating project incarnation #713 remains OPEN/P1 for deferred/apply-later Writer, Copilot, logline, proofread, synopsis/export, outline-preview and analogous retained-result authority.
• ai(lifecycle): unify Stop Generation, cancellation, supersession and duplicate-request semantics #714 remains OPEN/P1 for Stop Generation, cancellation, supersession, timeout/lifecycle and late-event semantics.

Therefore:

direct image/interview project-authority E2E
→ may consume current #719/#734 source behavior

deferred Apply/Insert/Replace/Select/Export cross-project E2E
→ wait for / land with the relevant #713 source slice

Stop/supersession lifecycle E2E
→ consume the admitted #714 contract; do not invent lifecycle semantics in test code

Admission evidence already landed

#744 is terminal deterministic browser evidence for the Browser-Ollama OFF/ON Settings admission boundary: disabled means network-inert; local requests are admitted only after explicit opt-in/user action. Treat that as existing evidence rather than adding a duplicate #710 journey. Real Ollama/runtime inference remains separate #743/#556/#714 qualification.

Current transport architecture

Provider/lifecycle/discovery code has been decomposed through #748/#759/#761. Product E2E should still mock only the external transport boundary, but re-fetch the current provider adapter/request path rather than assuming all traffic originates inside the historical monolithic aiProviderService.ts.

ProForge planning vs runtime

#727/#728/#730 are now merged planning/evidence PRs only. They do not make ProForge token accounting (#732), Claude budgets/streaming (#731), or structured outputs (#729) functionally qualified. When #710 adds a representative ProForge browser journey, assert only runtime contracts that have actually landed.

Test authority

#709 remains the exhaustive coverage/disposition owner; #736 owns required-lane architecture/locale-safe locator/scaling policy; #712 owns only selected production-bundle canaries. Keep browser evidence representative rather than duplicating lower-layer race matrices.

This issue remains OPEN/P1. The ownership update narrows stale dependencies; it does not reduce the required product-flow inventory.