Skip to content

Add atomic transaction API - #28

Merged
findolor merged 1 commit into
mainfrom
transaction-api
Jul 1, 2026
Merged

findolor merged 1 commit into
mainfrom
transaction-api

Conversation

@findolor

@findolor findolor commented Jun 26, 2026 •

Copy link
Copy Markdown
Collaborator

Dependent PRs

Motivation

Consumers that pass SQLiteWasmDatabase.query.bind(localDb) into SDK transaction batches currently express logical transactions as separate query() calls: BEGIN TRANSACTION, individual statements, then COMMIT. sqlite-web serializes individual worker messages, but it does not hold a transaction-level lock across multiple query() calls. With multiple handles/tabs sharing a database, those split transaction calls can interleave and SQLite can return cannot start a transaction within a transaction.

Solution

  • Add public SQLiteWasmDatabase.transaction(statements) JS API that accepts { sql, params? }[].
  • Send the whole statement list as one worker job through the coordinator and DB worker queue.
  • Execute the batch inside a worker-owned SQLite transaction, returning the last statement result and rolling back on failure.
  • Reject user-supplied transaction-control statements (BEGIN, COMMIT, END, ROLLBACK, SAVEPOINT, RELEASE) so the worker owns transaction scope.
  • Add browser integration coverage that verifies successful transaction execution, rollback, and the old split-BEGIN interleaving failure.
  • Retry OPFS sahpool deletion after worker termination so wipeAndRecreate handles Chrome briefly retaining OPFS handles.
  • Update the rain.math.float submodule and dependency graph for the latest float crate.
  • Update Rainix so CI uses a Rust toolchain new enough for revm 36.
  • Install the float submodule Soldeer dependencies in build-submodules before running forge build.
  • Fix the local bundle script to rewrite the current wasm-bindgen export form for embedded classic workers.

Checks

By submitting this for review, I am confirming I have done the following:

  • made this PR as small as possible
  • unit-tested any new functionality
  • linked any relevant issues or PRs
  • included screenshots (if this involves a front-end change)

Validation run:

  • cargo fmt --all
  • cargo check --tests -p sqlite-web-core
  • cargo check --tests -p sqlite-web
  • ./scripts/local-bundle.sh
  • bun run test (158 passed | 4 skipped)
  • npm run lint-format-check in svelte-test
  • nix develop -c rustc --version (rustc 1.94.0)
  • nix develop -c build-submodules
  • nix develop -c local-bundle
  • Manual NO_HEADLESS wasm browser test for sqlite-web-core (98 passed; 0 failed)
  • Manual NO_HEADLESS wasm browser test for sqlite-web (38 passed; 0 failed before OPFS retry; CI rerun pending)
  • git diff --check

Summary by CodeRabbit

  • New Features

    • Added an exported transaction(statements) API to run multiple SQL statements atomically.
    • Introduced batched SQL execution support end-to-end, including optional parameters for each statement.
  • Bug Fixes

    • Improved batch execution robustness across leader/follower coordinator behavior, including safer initialization-pending handling.
    • More reliable OPFS database recreation by retrying directory deletion.
  • Tests

    • Added/extended wasm and integration tests for batch timeout scaling, batch validation, rollback-on-failure, and transaction interleaving across concurrent clients.
  • Chores

    • Improved worker wasm-bindgen initialization for local bundling.

@coderabbitai

coderabbitai Bot commented Jun 26, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@findolor, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 45 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: c9f08018-836e-476b-acbc-fa79066a4138

📥 Commits

Reviewing files that changed from the base of the PR and between c13d19d and df6914e.

⛔ Files ignored due to path filters (4)
  • Cargo.lock is excluded by !**/*.lock
  • flake.lock is excluded by !**/*.lock
  • svelte-test/bun.lock is excluded by !**/*.lock
  • svelte-test/package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (10)
  • Cargo.toml
  • flake.nix
  • lib/rain.math.float
  • packages/sqlite-web-core/src/coordination.rs
  • packages/sqlite-web-core/src/database.rs
  • packages/sqlite-web-core/src/messages.rs
  • packages/sqlite-web/src/db.rs
  • scripts/local-bundle.sh
  • svelte-test/package.json
  • svelte-test/tests/integration/transaction-interleaving.test.ts

Walkthrough

Adds atomic batch SQL execution end-to-end: new message types, coordinator routing for batch requests, transactional batch execution in the database layer, and a WASM-exported transaction method with input normalization. Also updates workspace dependencies, submodule build steps, and worker bundle initialization.

Changes

Batch transaction support

Layer / File(s) Summary
Batch message contracts
packages/sqlite-web-core/src/messages.rs
SqlBatchStatement and batch request/worker variants are added with serialization coverage.
Coordinator routing and dispatch
packages/sqlite-web-core/src/coordination.rs
Batch requests are routed through leader/follower paths, payload-based DB jobs are introduced, and queue handling switches to DbJobPayload.
Batch execution and validation
packages/sqlite-web-core/src/database.rs
exec_batch runs statements in a transaction, rejects transaction-control SQL, and is covered by wasm tests for rollback and keyword handling.
WASM transaction API
packages/sqlite-web/src/db.rs, svelte-test/tests/integration/transaction-interleaving.test.ts, svelte-test/package.json
transaction is exported to JS, batch inputs are normalized, OPFS deletion retries are added, and integration tests cover success, rollback, isolation, and follower execution.

Tooling updates

Layer / File(s) Summary
Workspace and submodule inputs
Cargo.toml, flake.nix, lib/rain.math.float
Dependency specs, the submodule pin, and the Forge submodule build sequence are updated.
Worker bundle init
scripts/local-bundle.sh
Worker initialization is rewritten to use self.wasm_bindgen(...) in the worker scope.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 63.79% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately summarizes the main change: adding an atomic transaction API for batched SQL execution.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch transaction-api

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copy link
Copy Markdown
Collaborator Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

@findolor findolor self-assigned this Jun 26, 2026
@findolor
findolor force-pushed the transaction-api branch 5 times, most recently from c959ab8 to 0f66c72 Compare June 26, 2026 11:08
@findolor
findolor requested review from 0xgleb and JuaniRios June 26, 2026 11:21
Comment thread packages/sqlite-web-core/src/database.rs
Comment thread packages/sqlite-web-core/src/database.rs Outdated
Comment thread svelte-test/tests/integration/transaction-interleaving.test.ts Outdated
Comment thread packages/sqlite-web-core/src/database.rs
Comment thread packages/sqlite-web-core/src/database.rs
Comment thread packages/sqlite-web-core/src/coordination.rs Outdated
Comment thread packages/sqlite-web-core/src/database.rs
Comment thread packages/sqlite-web-core/src/database.rs
Comment thread packages/sqlite-web-core/src/messages.rs
Comment thread packages/sqlite-web-core/src/database.rs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@Cargo.toml`:
- Line 52: The `wasm-bindgen-utils` dependency pin is mismatched with the
current `prelude` imports, since
`wasm_bindgen_utils::prelude::serde_wasm_bindgen` is not available in the
crates.io release. Update the affected call sites to import and use
`serde_wasm_bindgen` directly, or keep the existing git-pinned revision of
`wasm-bindgen-utils` so the `prelude` paths used by the relevant modules
continue to compile.

In `@flake.nix`:
- Around line 63-66: The Foundry setup in the task is running from the repo root
instead of the rain.math.float submodule, so update the sequence in the flake
task to change into lib/rain.math.float before invoking any forge commands.
Ensure the soldeer dependency install happens first, then run forge install and
forge build from that directory so the commands target the correct project and
use its local Foundry config.

In `@packages/sqlite-web/src/db.rs`:
- Around line 122-123: The params extraction in `Db::...` is swallowing
`Reflect::get` failures by converting them to `undefined`, unlike the `sql`
getter path. Update the `params` property access to propagate the error instead
of defaulting to `JsValue::UNDEFINED`, using the same error-handling style as
the existing `sql` getter logic so a throwing `params` getter does not execute
silently.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: fe61f091-285f-490c-a549-13cf7e1ef03b

📥 Commits

Reviewing files that changed from the base of the PR and between a4d6dbe and 118472a.

⛔ Files ignored due to path filters (4)
  • Cargo.lock is excluded by !**/*.lock
  • flake.lock is excluded by !**/*.lock
  • svelte-test/bun.lock is excluded by !**/*.lock
  • svelte-test/package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (10)
  • Cargo.toml
  • flake.nix
  • lib/rain.math.float
  • packages/sqlite-web-core/src/coordination.rs
  • packages/sqlite-web-core/src/database.rs
  • packages/sqlite-web-core/src/messages.rs
  • packages/sqlite-web/src/db.rs
  • scripts/local-bundle.sh
  • svelte-test/package.json
  • svelte-test/tests/integration/transaction-interleaving.test.ts

Comment thread Cargo.toml
Comment thread flake.nix Outdated
Comment thread packages/sqlite-web/src/db.rs Outdated
@findolor
findolor requested a review from 0xgleb June 30, 2026 13:12
Comment thread packages/sqlite-web-core/src/database.rs
Comment thread packages/sqlite-web-core/src/coordination.rs
Comment thread svelte-test/tests/integration/transaction-interleaving.test.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/sqlite-web-core/src/database.rs`:
- Around line 927-929: The SQL filtering in `database.rs` is incorrectly
treating any string where `Self::first_sql_keyword(sql)` returns `None` as
trivia-only, which skips executable statements prefixed by a delimiter and can
also hide invalid transaction-control input. Update the logic around the
`first_sql_keyword` check so it distinguishes true comment/whitespace-only SQL
from non-trivia statements that merely lack a leading keyword, and make the
transaction path fail with an error and rollback instead of continuing silently.
Use the existing `Self::first_sql_keyword` and surrounding transaction execution
flow to route delimiter-prefixed SQL into either normal execution or an explicit
rejected-statement error.
- Around line 944-949: The batch execution logic is counting affected rows from
`statement_result` even when the statement returns rows, which can pick up stale
`sqlite3_changes()` from prior DML. Update the handling around
`exec_prepared_statement` so only non-query statements contribute to
`total_affected_rows`, and make sure `last_statement_rows`/`affected` are
separated by whether the statement actually returned rows. Use the existing
`statement_result` match in `database.rs` to gate the increment and avoid
carrying over changes from SELECT-like statements.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 8fef518b-b173-4c55-a863-f3b31559057a

📥 Commits

Reviewing files that changed from the base of the PR and between 118472a and c13d19d.

⛔ Files ignored due to path filters (4)
  • Cargo.lock is excluded by !**/*.lock
  • flake.lock is excluded by !**/*.lock
  • svelte-test/bun.lock is excluded by !**/*.lock
  • svelte-test/package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing (10)
  • Cargo.toml
  • flake.nix
  • lib/rain.math.float
  • packages/sqlite-web-core/src/coordination.rs
  • packages/sqlite-web-core/src/database.rs
  • packages/sqlite-web-core/src/messages.rs
  • packages/sqlite-web/src/db.rs
  • scripts/local-bundle.sh
  • svelte-test/package.json
  • svelte-test/tests/integration/transaction-interleaving.test.ts

Comment thread packages/sqlite-web-core/src/database.rs
Comment thread packages/sqlite-web-core/src/database.rs
@findolor
findolor requested a review from JuaniRios June 30, 2026 15:00
@findolor
findolor merged commit 0c9a50c into main Jul 1, 2026
4 checks passed
@findolor findolor mentioned this pull request Sep 28, 2026
8 tasks done
findolor added a commit that referenced this pull request Sep 30, 2026
## Related PRs and issue

- [RAI-2650](https://linear.app/makeitrain/issue/RAI-2650/add-bounded-atomic-sql-dump-import-to-sqlite-web-and-release-it)
- Producer grouped SQL change: [raindex#2893](rainlanguage/raindex#2893). This PR adds an upstream import API for the subsequent browser integration; neither PR needs the other to merge.
- Builds on the existing `transaction(statements)` API from [sqlite-web#28](#28).

## Motivation

The browser bootstrap currently materializes Rust and JavaScript statement objects for the SQL dump and sends the array through one transaction callback. Grouping rows on the producer side reduces this work, but the browser still needs a bounded way to stream SQL text into one atomic import without exposing partial data across tabs.

## Solution

- Add `beginSqlDumpImport`, `appendSqlDumpChunk`, `finishSqlDumpImport`, and `cancelSqlDumpImport` to the public wasm API. The database worker owns one `BEGIN IMMEDIATE` transaction across chunks and commits only after a successful finish; SQL errors, invalid chunks, cancellation, and abandoned sessions roll back.
- Parse SQL incrementally across chunk boundaries, including strings, comments, and trigger bodies. Bound each UTF-8 chunk to 512 KiB and an unfinished statement to 16 MiB. Reject row-returning statements during import to avoid collecting unbounded results.
- Reject connection PRAGMA settings, `EXPLAIN`, and `ATTACH`/`DETACH` before preparation so failures cannot leave connection changes that transaction rollback would not undo. Accept and skip the standard `PRAGMA foreign_keys=OFF;`/`=0` dump header while preserving existing foreign-key enforcement.
- Reject unpaired UTF-16 surrogates before conversion or dispatch. A cached JavaScript regex validates each chunk with one call across the Wasm boundary.
- Run all core import tests against SQLite's memory VFS without silently skipping failed opens, alongside real OPFS browser integration. Cover partial lexical states, triggers, statement limits, rollback, cancellation, and caller-visible leader-loss outcomes.
- Route import actions through the existing leader/follower coordinator. Reject competing queries and transactions while an import is active, limit outstanding import work per client instance, and expire abandoned imports after 120 seconds of inactivity on the next request or 30-second watchdog tick. An import response is reported only after the database worker resolves it, so a delayed commit cannot be reported as a follower timeout.
- Document the API and limits; add browser integration tests and an isolated synthetic benchmark.

This PR does not change the producer, the browser bootstrap call site, the dump format, or the published package version. The repository's main-branch release workflow publishes the next patch version after merge.

## Checks

- [x] `nix develop -c build-submodules`
- [x] `nix develop -c local-bundle`
- [x] `nix develop -c rainix-rs-static`
- [x] Core WASM tests in Chrome: 124 passed (15 core import tests execute through the memory VFS)
- [x] Public WASM tests in Chrome: 42 passed
- [x] `nix develop -c npm test` in `svelte-test`: 170 passed, 4 existing skips
- [x] `nix develop -c npm run lint-format-check` in `svelte-test`
- [x] `git diff --check`

The latest WASM suites passed under the standard `nix develop -c test-wasm` wrapper with a matched Chrome for Testing / ChromeDriver 147 pair configured locally. The initial run with the system ChromeDriver 144 failed before tests started; the matched pair passed all tests. The packaged browser integration suite passed under Playwright Chromium.

`nix develop -c npm run test:benchmark` also passed. Its three sequential cases import 10,000 equal rows: single-row transaction including construction 59.6 ms, grouped transaction 12.4 ms, and identical grouped SQL through import chunks 15.9 ms. This supersedes the earlier two-case comparison: grouping and the import API must be measured separately. This is a fixed-order smoke measurement with warm storage and small chunks, not an end-to-end or production speedup estimate; download, large-dump object allocation, and indexes require raindex benchmarks.

Review focus: cross-tab serialization, transaction-marker handling, containment of connection state, and rollback behavior when a chunk or finish fails. A leader change before an import response reports an unknown outcome to the caller. Worker failure during finish can leave the outcome unknown even when an error response arrives; check/reset before retrying. Any lost finish response requires checking/resetting the database before retrying; the new test deliberately stalls a DB-worker response to make that path deterministic.

Local Codex review completed three initial rounds with four reviewers, followed by two follow-up rounds with three reviewers, with no OpenCode supplement. Three simplification passes checked the follow-up fixes. All seven latest review comments are addressed: skipped headers are excluded from counts, empty dumps are consistently rejected, unmatched markers have a specific error, the unused queue guard is removed, benchmark APIs use identical grouped SQL, and the supported dumps and liveness/unknown-outcome limitations are documented. The earlier fixes also close the `EXPLAIN PRAGMA` bypass and prevent attachment changes from surviving cancellation.



<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

* **New Features**
  * Added support for importing SQL dumps in chunks, with controls to begin, finish, or cancel an import across connected clients.
  * Imports validate chunk encoding and SQL statements, reject unsupported operations, and roll back on errors or after 120 seconds of inactivity. Regular database operations are blocked while an import is active.
* **Documentation**
  * Added guidance on streaming SQL dumps, import constraints, failure handling, and checking the database before retrying when the commit outcome is unknown.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants