Skip to content

analytics(duckdb): requalify default-on DuckDB-WASM runtime, privacy, persistence and recovery semantics #716

Open
Open
analytics(duckdb): requalify default-on DuckDB-WASM runtime, privacy, persistence and recovery semantics#716
Labels
P2

Description

@qnbs
qnbs
opened on Sep 11, 2026

Context

enableDuckDbAnalytics is currently a default-on, experimental production feature. It is not just a hidden developer toggle: current code uses DuckDB-WASM for analytics queries, RAG vector mirroring/querying, local AI telemetry, Reference Panel querying, ProForge analytics/history paths, and related worker-backed storage behavior.

Current production architecture already contains substantial hardening:

  • feature-gated initialization in useDuckDb;
  • retries/timeouts and degraded unavailable state;
  • OPFS→memory fallback reporting;
  • WorkerBus-backed DuckDB execution;
  • the central app/analyticsGate.ts privacy authority;
  • Settings → Privacy → Analytics opt-out;
  • vector-only RAG mirroring with manuscript text retained outside DuckDB;
  • previous SEC-6 hardening of persisted prose columns.

However, there is no dedicated open issue owning the whole default-on runtime contract. Current tests are mostly unit/service-level and repository guidance explicitly mocks services/duckdb/duckdbClient rather than initializing real DuckDB-WASM in normal tests. Deep feature-flag coverage is currently only advisory and mostly boot/settings smoke.

This matters because WorldScript has already experienced a real production-only failure class where the advertised local WASM stack (including DuckDB-WASM) was broken by deployed CSP while green CI did not catch it.

This issue owns a bounded DuckDB product/runtime requalification and hardening pass. It does not replace #482 (general IDB/OPFS storage architecture), #549 (security/privacy truth), #550 (CI cost), #709 (coverage authority), or #712 (production-bundle canaries).

Goal

Establish and prove this invariant:

When DuckDB analytics is enabled, WorldScript initializes and uses DuckDB only within the admitted privacy/storage/runtime contract, degrades safely when unavailable, remains project-isolated, and never silently converts a browser/runtime/storage limitation into data loss or misleading success.

1. Inventory every production DuckDB caller

Derive the complete call graph from current source.

At minimum inspect:

  • hooks/useDuckDb.ts;
  • hooks/useAnalytics.ts;
  • services/duckdb/duckdbClient.ts;
  • services/duckdb/duckdbAnalytics.ts;
  • services/duckdb/duckdbSchema.ts;
  • WorkerBus DuckDB worker/pool registration;
  • app/analyticsGate.ts;
  • AI telemetry service;
  • RAG vector mirror/query paths;
  • Reference Panel DuckDB query UI;
  • ProForge analytics/history/memory paths;
  • any listener middleware writes;
  • cleanup/reset/factory-reset paths;
  • PWA/Tauri production-build assets/config.

Classify each caller:

READ_ONLY_ANALYTICS
LOCAL_TELEMETRY_WRITE
RAG_VECTOR_WRITE
RAG_VECTOR_QUERY
USER_QUERY_UI
PROFORGE_ANALYTICS
MAINTENANCE / MIGRATION
RESET / DELETE

No production write path should remain outside the central privacy gate without explicit justification.

2. Privacy authority

Preserve one authoritative persistence decision:

enableDuckDbAnalytics
AND
settings.privacy.analyticsEnabled
AND
any narrower data-class policy

Audit all writes, including asynchronous and deferred writes, for TOCTOU behavior.

Required guarantees:

  • turning Privacy → Analytics OFF prevents subsequent persistent analytics writes;
  • a stale boolean captured before an async operation cannot authorize a later write after opt-out;
  • feature-flag OFF prevents DuckDB analytics persistence and background initialization as defined;
  • telemetry fallback storage obeys the same user-facing privacy intent;
  • no manuscript prose is newly introduced into DuckDB merely to simplify analytics;
  • existing SEC-6 encrypted prose treatment remains intact where applicable;
  • no cloud transmission is implied by local analytics.

Do not weaken the privacy gate to make analytics tests easier.

3. Initialization lifecycle

Audit the complete initialization state machine:

idle
→ initializing
→ ready
OR
→ memory fallback / unavailable / error

Verify:

  • concurrent callers do not initialize multiple DuckDB runtimes/pools;
  • retries are bounded;
  • timeout cleanup is correct;
  • abort/unmount does not leave hanging worker tasks/listeners;
  • feature disable during initialization settles deterministically;
  • re-enable can recover cleanly where intended;
  • WorkerBus pool respawn/re-registration restores a usable DuckDB connection;
  • page reload does not inherit impossible in-memory status from a prior runtime.

Do not solve lifecycle races with broad retries alone.

4. OPFS / in-memory fallback contract

DuckDB currently falls back when OPFS is unavailable.

Make the behavior explicit and tested:

OPFS available
→ persistent analytics database

OPFS unavailable / blocked
→ in-memory DuckDB only
→ clear truthful UI/status
→ app functionality continues

Verify:

  • no authoritative project data depends exclusively on DuckDB persistence;
  • memory fallback does not falsely claim persistence across reload;
  • reload after memory fallback is safe;
  • quota/security/permission errors are classified truthfully;
  • unsupported browsers degrade without breaking writing/export/core flows.

General storage hierarchy/quota UX remains #482.

5. Schema/migration safety

Audit current DDL + migration sequence.

Required properties:

  • migrations are idempotent or explicitly versioned;
  • repeated startup cannot corrupt schema;
  • interrupted migration leaves a recoverable state;
  • future/unknown schema state fails safely;
  • table/index evolution does not silently discard local analytics records;
  • migration failure does not break the rest of the app;
  • privacy-sensitive columns retain their admitted protection semantics.

If the current DDL model intentionally uses additive idempotent statements rather than a schema-version table, document that contract and prove it with fixtures.

Do not migrate user data destructively merely to simplify tests.

6. Project isolation

DuckDB analytics/RAG tables must not mix projects accidentally.

Verify every query/write includes project ownership where required.

Regression cases should include:

project A and B share section/entity IDs
switch A → B
reset/import/restore
same nominal project id with a new incarnation where relevant
RAG rebuild/query after project switch
analytics queries while previous request is still in flight

This issue owns DuckDB data partition/query correctness, not #707/#713 AI result ownership.

If analytics records intentionally aggregate across projects, make that explicit instead of relying on missing filters.

7. RAG vector mirror/query contract

Current hybrid RAG can mirror vectors to DuckDB and use DuckDB similarity ranking.

Verify:

  • only admitted vector/metadata classes are mirrored;
  • project ID + section/chunk identity are stable;
  • vector dimensionality/schema mismatch fails safely;
  • stale/rebuilt indexes do not produce cross-project results;
  • DuckDB query failure degrades to the intended IDB/CPU path where product semantics allow;
  • feature/privacy disable prevents new mirror writes;
  • index rebuild honors privacy changes during async embedding work;
  • deleting/resetting project state does not leave misleading analytics/RAG records presented as current.

Coordinate broader Local-First/project-lifecycle cleanup with existing storage owners where needed.

8. Query safety

For user-facing or dynamic DuckDB queries:

  • use parameters rather than concatenating untrusted values;
  • validate/limit any raw SQL surface exposed to user-facing tooling;
  • do not allow arbitrary SQL to mutate protected/internal tables unless intentionally designed;
  • surface query errors without leaking manuscript content or filesystem paths;
  • bound result sizes and expensive scans where appropriate.

Do not turn this issue into a general SQL IDE redesign.

9. WorkerBus lifecycle

DuckDB now routes through WorkerBus v2.

Audit:

  • pool creation/termination;
  • worker crash/respawn;
  • connection restoration after respawn;
  • parameter forwarding;
  • cancellation/abort;
  • queued task settlement during terminate/reset;
  • circuit-breaker/dead-letter interaction;
  • stale response routing.

#552 remains the broader WorkerBus/Core extraction owner. This issue owns the DuckDB consumer contract only.

10. Default-on feature semantics

Because enableDuckDbAnalytics defaults ON, #709 must not classify it as covered merely because default-state E2E boots successfully.

Recommended #709 disposition:

REQUIRED functional/runtime contract at lower layers
+
REQUIRED flag OFF-state behavior
+
small browser product smoke where stable
+
selected production-bundle real-WASM canary under #712 if unique signal exists

Changing the flag default in the future must not be the only thing deciding whether DuckDB gets exercised.

11. Real DuckDB-WASM qualification

Normal unit tests may continue mocking DuckDB for speed and determinism, but the repository needs at least one controlled real-runtime layer.

Prefer a small bounded qualification test that actually loads the self-hosted DuckDB-WASM artifact and proves:

initialize
schema bootstrap
parameterized insert
parameterized query
basic persistence mode classification
shutdown/cleanup

For browser production qualification, #712 may run one canary against the exact built dist if this catches a unique CSP/WASM/chunk/base-path class.

Do not run the full analytics suite against real DuckDB in every unit-test shard.

12. CSP / production-build regression

Given the historical production-only WASM CSP outage, explicitly verify that current production policy permits the exact DuckDB runtime mechanism used today.

Do not assume a generic WebAssembly.instantiate() smoke fully proves:

  • worker loading;
  • DuckDB bundles;
  • pthread/worker behavior if used;
  • OPFS access;
  • actual DuckDB initialization.

#712 owns the production-canary infrastructure. This issue defines the DuckDB behavior that canary should prove if selected.

Never weaken CSP merely to make DuckDB work; adapt the implementation to the admitted CSP where possible.

13. Performance/resource behavior

Measure, don't guess.

Capture representative:

  • cold init time;
  • warm init time;
  • memory overhead;
  • RAG query latency;
  • analytics query latency;
  • large-project behavior;
  • worker restart cost;
  • OPFS database size growth.

Define reasonable regression evidence, not brittle absolute CI timing gates.

#550 owns pipeline cost; this issue owns product/runtime performance evidence.

14. Reset / cleanup / remanence

Inventory what happens on:

  • project delete/reset;
  • factory reset;
  • analytics opt-out;
  • feature flag OFF;
  • encryption transitions;
  • storage cleanup.

Differentiate:

STOP_FUTURE_WRITES
DELETE_EXISTING_ANALYTICS
PRESERVE_EXISTING_LOCAL_DATA
CLEAR_REGENERABLE_RAG_INDEX

Do not silently delete existing local analytics solely because the feature is temporarily disabled unless product policy explicitly requires it.

Coordinate actual browser-storage deletion mechanics with #482/#602 where overlapping.

15. UX truth

User-visible state must distinguish:

ready + persistent
ready + memory-only
initializing
unavailable
disabled by feature flag
disabled by privacy preference

Do not present an empty chart as if it means “zero activity” when DuckDB is actually unavailable.

Do not expose low-level WASM/OPFS implementation details unless useful for diagnostics.

Relationship to other issues

Acceptance criteria

  • Every production DuckDB caller/read/write path is inventoried and classified.
  • All persistent writes are reconciled with the central analytics/privacy gate.
  • Privacy opt-out cannot be bypassed by stale async-captured state.
  • DuckDB initialization/retry/abort/re-enable lifecycle is deterministic.
  • OPFS persistent vs memory fallback semantics are truthful and tested.
  • Schema/bootstrap/migration is repeatable and failure-safe.
  • Project isolation is proven for analytics and RAG records, including ID-collision fixtures where relevant.
  • RAG vector mirror/query has explicit fallback and privacy semantics.
  • Worker crash/respawn/terminate behavior cannot strand DuckDB tasks or permanently poison the client state.
  • At least one controlled real DuckDB-WASM qualification path exists beyond mocks.
  • A production-bundle DuckDB canary is added under test(prod-e2e): add risk-based functional canaries against the built production bundle #712 if it provides unique CSP/WASM/worker signal.
  • Default-on status does not substitute for intentional test coverage under test(feature-flags): establish exhaustive coverage authority and prevent opt-in E2E blind spots #709.
  • UI distinguishes persistent/memory/unavailable/disabled states truthfully.
  • cleanup/reset/remanence behavior is explicitly classified and coordinated with storage owners.
  • no cloud transmission or new plaintext manuscript persistence is introduced.
  • focused tests, pnpm run ci:prepush, exact-head CI/CD and CodeQL are green before implementation PR merge.

Non-goals

Priority

P2 default-on runtime hardening.

Escalate to P1 if the requalification confirms any of:

privacy opt-out bypass
cross-project query/write leakage
persistent schema corruption/data loss
production DuckDB currently broken under deployed bundle/CSP

Recommended sequencing:

#709 coverage authority classification
        ↓
THIS ISSUE source/runtime requalification
        ↘
         #712 selected real-WASM production canary
        ↓
#623 evidence-based graduation if any advisory scenario becomes required

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions