Skip to content

fix(storage): converge concurrent fresh WAL initialization - #2003

Merged
Astro-Han merged 1 commit into
apache:mainfrom
zhiiw:codex/sqlite-first-open-wal-race
Aug 3, 2026
Merged

fix(storage): converge concurrent fresh WAL initialization#2003
Astro-Han merged 1 commit into
apache:mainfrom
zhiiw:codex/sqlite-first-open-wal-race

Conversation

@zhiiw

@zhiiw zhiiw commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

  • make fresh operational-state SQLite initialization converge when multiple processes race to enable WAL
  • retry only transient SQLITE_BUSY / locked failures within the existing 5-second initialization bound
  • add a production-shaped, 12-round, two-process regression through acquireOperationalStateDatabase()

Invariant and boundary

  • Owner: the operational-state SQLite initializer
  • Atomicity boundary: SQLite's persistent journal-mode transition; each process re-observes the mode before attempting the transition
  • Failure state: transient lock contention is retried; an unsupported mode or contention beyond 5 seconds still fails closed
  • Rollback: revert this commit; there is no schema, protocol, or persisted-data migration

Compatibility

  • rebuilt from current upstream/main (72027d2bb at implementation time)
  • the same patch was applied to PR Seal managed workspace baseline admission #1872's head, preserving its schema-8 storage-root binding; the production-shaped fresh-open test passed there as well

Verification

  • RED before the production change: the first fresh-open round failed with database is locked
  • focused schema / operational-owner / recovery concurrency suite: 21 passed
  • fresh operational-owner race: 12/12 rounds passed on current main
  • fresh operational-owner race: 12/12 rounds passed with Seal managed workspace baseline admission #1872
  • @maka/storage build and formatting checks passed

The full storage suite was also attempted on local Windows. Unrelated tests hit the repository's existing parallel temp-cleanup EBUSY failures while deleting open runtime.sqlite[-shm] files; the focused tests above completed cleanly.

Fixes #1544

中文说明

变更摘要

  • 修复多个进程同时首次打开同一个 operational-state 空数据库时,竞争设置 WAL 导致 database is locked 的问题
  • 只对 SQLITE_BUSY / locked 做有界重试,沿用 5 秒初始化上限
  • 新增真实生产入口测试:两个独立进程经 acquireOperationalStateDatabase() 同时起跑,共重复 12 轮

不变量与边界

  • Owner: operational-state SQLite initializer
  • 原子性边界: SQLite 持久化的 journal-mode 转换;每个进程在尝试转换前重新读取当前模式
  • 失败状态: 短暂锁竞争会重试;超过 5 秒或得到不支持的模式时仍然 fail-closed
  • 回滚方式: 直接回滚本提交;本 PR 不修改 schema、runtime protocol 或持久化数据格式

兼容性验证

测试

  • 修改生产代码前,新增测试第一轮即复现 database is locked
  • schema / operational owner / recovery concurrency 定向套件:21 项通过
  • 最新 main:12/12 轮 fresh-open 通过
  • Seal managed workspace baseline admission #1872 组合分支:12/12 轮 fresh-open 通过
  • storage build 与格式检查通过

本机 Windows 也尝试了完整 storage suite;若干无关测试在并行删除仍被占用的 runtime.sqlite[-shm] 临时文件时触发既有 EBUSY,不属于本次 WAL 初始化回归。上述定向测试均已通过。

@zhiiw
zhiiw marked this pull request as ready for review August 3, 2026 11:41

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: MERGE-READY ✅ (independent deepseek-v4-flash review)

Reviewed the full diff (+103/−18, 3 files, all in packages/storage) and reproduced the race locally.

  • Fix verified effective: on main's old logic (busy_timeout + single journal-mode switch), a two-process probe failed 3/12 rounds with database is locked; with the PR's ensureWalJournalMode + bounded 5s retry, 0/12 failures, convergence ≤269ms. The mechanism matches the repo's existing precedent from #1770 (sqlite-long-term-memory-schema.ts ensureWalJournalMode), and journal-mode conversion is serialized by SQLite's header lock — no corruption/lost-update window; fail-closed preserved (non-BUSY errors throw immediately).
  • Test quality good: new real-child-process × production entry (acquireOperationalStateDatabase()) test with file-barrier sync, 12 rounds, asserting both process exit codes, journal_mode='wal', user_version=8; passes on CI (test_workspaces). No schema/protocol change, clean single-commit revert.
  • Storage jobs all green (test_workspaces, test_runtime_host, test_headless, typecheck).

Non-blocking P3 (optional): (1) ensureWalJournalMode is now duplicated in two schema files (maintenance-drift risk — consider a shared helper or cross-referencing comments); (2) the race test is probabilistic and doesn't assert that a round actually hit contention; (3) the 'memory' early-return is a dead branch for file DBs.

Merge prerequisite: the only red check (e2e_shard (1/2)plan-reminders.spec.ts:86 toBeFocused()) is a known pre-existing flake: the same test fails identically on main (run 30811251917), the repo's playwright.config.ts:13-16 documents toBeFocused() failing on window-blur with retries: 0 by design, and this PR touches zero desktop code (shard 2/2 is green). Please re-run the e2e job (or apply the repo's established flake handling) before merging; the PR itself needs no changes.

@zhiiw

zhiiw commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

I reviewed the approval and reran the failed E2E shard as requested. The rerun reproduced the exact same pre-existing desktop focus flake:

  • e2e/plan-reminders.spec.ts:86
  • expect(...).toBeFocused()
  • expected focused, received inactive
  • 30 other tests in the shard passed

This PR changes only three storage files, and all storage/runtime/typecheck checks remain green. I am leaving the three optional P3 observations out of this PR so the WAL initialization fix keeps a single invariant and a clean revert boundary.

No production change is warranted in #2003 for this rerun result; the remaining red status is the repository's existing UI focus flake.

中文说明

已按 review 建议重跑失败的 E2E shard,结果再次复现完全相同的既有 Desktop 焦点 flaky:

  • e2e/plan-reminders.spec.ts:86
  • expect(...).toBeFocused()
  • 期望 focused,实际 inactive
  • 同一 shard 其余 30 个测试通过

本 PR 只修改三个 storage 文件,storage/runtime/typecheck 检查仍全部通过。三个可选 P3 不并入本 PR,以保持 WAL 初始化修复只有一个主要不变量和清晰回滚边界。

因此此次重跑结果不需要修改 #2003 的生产代码;剩余红灯来自仓库既有 UI focus flaky。

@Astro-Han

Copy link
Copy Markdown
Contributor

Merged — thanks for the fix!

Heads-up on why the CI was red: it was never this PR's code. The branch was based on a pre-fix main snapshot (11:39, before #2001 merged at 11:58), so the PR's CI kept running the stale merge-ref — e2e_shard (1/2) failed on plan-reminders.spec.ts:86 (toBeFocused on the edit-dialog title), a known autofocus regression from #2002 that #2001 (f7cb2c3b8) already fixed on main. Squashing onto latest main re-bases the change onto the fixed code, so main's CI is green again.

One optional follow-up from the independent review (pre-existing, not blocking this PR): the same concurrent first-open WAL race still lives in configureSqliteSessionMetadataDatabase (sqlite-session-metadata-schema.ts:834-837 — plain busy_timeout + journal_mode with no retry, reachable via the readUsageStats IPC chain). Applying the same ensureWalJournalMode pattern there would close it — happy to file an issue if you want to pick it up.

@Astro-Han
Astro-Han merged commit 1705111 into apache:main Aug 3, 2026
45 of 55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(storage): serialize concurrent first-open WAL initialization

2 participants