Filesystem Storage for Parallel AI Agents
Website · Quick start · How it works · Benchmarks · Milestones
LayerFS gives Ephemeral AI Computer a durable workspace layer where agents can read and edit files independently, share unchanged content, and publish changes through one authoritative SQLite-backed workspace.
| Shared content | Local edits | Durable publication |
|---|---|---|
| SHA-256 CAS objects and authenticated manifests keep unchanged bytes shared. | FastCDC and bounded local-rebuild paths keep edits local when the workload allows it. | SQLite transactions, leases, revisions, quotas, and conflict checks protect the authoritative workspace. |
The C3 storage pipeline described in Part III of the Agent Infra Book is represented directly in this repository.
An edit follows this path:
- Split changed bytes into content-defined chunks.
- Store new chunks by hash and reuse existing CAS objects.
- Build and authenticate a new ordered manifest.
- Commit revision, lease, quota, and head changes atomically in SQLite.
- Read from a branch head or publish after verifying the expected base revision.
| C3 mechanism | Repository implementation | Status |
|---|---|---|
| CAS: share unchanged bytes | packages/fs/src/cas/ and packages/fs/src/sqlite/content-repository.ts |
✅ M1/M2 accepted |
| CDC: reconnect after local edits | packages/fs/src/cdc/fastcdc.ts and packages/fs/src/operations/local-rebuild.ts |
✅ M3 durable local-rebuild path |
| Authenticated manifests | packages/fs/src/manifests/ and packages/fs/src/sqlite/manifest-tree-repository.ts |
✅ M2 accepted |
| COW pages and patches | packages/fs/src/cow/pages.ts and packages/fs/src/sqlite/overlay-repository.ts |
✅ M4 branch overlays and bounded publication |
| Branch bases and conflict checks | packages/fs/src/branches/, branch-repository.ts, and branch-engine.ts |
✅ M4 accepted |
| One durable authority | packages/fs/src/sqlite/driver.ts and transaction-scoped repositories |
✅ M2 accepted |
The result is a filesystem foundation for C3-style execution: shared immutable data, private agent state, bounded edits, and explicit publication into one durable main view.
Full workspace copies make branch creation and repeated edits scale with workspace size. LayerFS separates immutable content from workspace state instead:
- CAS shares identical content across files, revisions, and agents.
- CDC helps insertions and deletions reconnect with unchanged content.
- Merkle manifests provide authenticated, ordered file content.
- SQLite supplies transactions, quotas, leases, and recovery boundaries.
- Explicit fallbacks preserve correctness when an edit exceeds a bounded local path.
The system may use a streamed rebuild for a large or unsupported edit, but it must not trust a stale derived index or return incorrect bytes.
M0 foundation ✅
M1 content engine ✅
M2 SQLite storage ✅
M3 filesystem I/O ✅
M4 branches ✅
M5 maintenance ✅
M6 Cloudflare parity ✅
M7 Node VFS ✅ latest accepted milestone
M8–M10 integration ⏳
| Milestone | Scope | Status |
|---|---|---|
| M0 | Repository and test foundation | ✅ Accepted |
| M1 | CAS, CDC, COW, patches, and manifests | ✅ Accepted |
| M2 | Transactional SQLite storage and Node driver | ✅ Accepted |
| M3 | Filesystem namespace, revisions, and I/O | ✅ Accepted |
| M4 | Branches and publication | ✅ Accepted |
| M5 | Maintenance, recovery, and bounded scale | ✅ Accepted |
| M6 | Cloudflare Durable Object SQLite parity | ✅ Accepted |
| M7 | Node VFS and real mounted FUSE | ✅ Latest accepted |
| M8–M10 | Replication, release, and Computer integration | ⏳ In progress |
M6 adds the faithful local Cloudflare Durable Object adapter and runtime suite using
ctx.storage.sql and transactionSync, including real runtime eviction,
released-schema migration, exhaustive filesystem/publication/maintenance fault matrices,
the unchanged 100,000-row scale fixture, and an exact deployable preview bundle. Both
accepted target selections complete within their independent 600-second budgets without
Cloudflare credentials, deployment, network access, or external state.
See the implementation plan, M6 exit record, and M6 handoff.
M7 adds the synchronous Node VFS provider, opaque core bridge, coordinated namespace and inode semantics, bounded multi-edit COW, fault/resource coverage, and the exact real-kernel FUSE profile. Candidate-bound evidence records 23 local tests and the full 9,056-operation mounted profile in 24.8 seconds. See the M7 evidence and M7 handoff.
The accepted Linux x64 real-FUSE run used a 16 MiB deterministic payload with SQLite on
tmpfs. Operating-system cache dropping was unavailable, so the restart read below is
not presented as a guaranteed cold-cache result.
| Mounted operation | Workload | Accepted time |
|---|---|---|
Initial write and fsync |
16 MiB | 1,134.418 ms (14.1 MiB/s) |
| Full read and SHA-256 after restart | 16 MiB | 101.370 ms (157.8 MiB/s) |
| Full-file materialization | Same mounted read after restart | 101.370 ms |
COW edits and final fsync |
5,000 one-byte write callbacks | Not timed as one phase |
| Complete mounted profile | 9,056 operations and 3 restarts | 24,767 ms |
The evidence records every one-byte edit callback and one successful flush. Individual edit calls were below 27.266 ms, the cutoff of the retained ten slowest operations, but the run did not retain an aggregate edit-phase time or edit p50/p95. See the raw real-FUSE log for the exact environment, resource peaks, digests, and operation counts.
The mini-benchmark measures the file-backed Node SQLite engine directly. It does not include FUSE, page-cache effects, or the complete Computer execution path. The tables compare the pre-improvement baseline, the accepted M2 implementation, and the accepted M3 implementation.
| Metric | Baseline | M2 | M3 |
|---|---|---|---|
| Cold 100 MiB write | 17.9 MiB/s | 44.2 MiB/s | 60.0 MiB/s |
| Cold 100 MiB read | 43.8 MiB/s | 118.1 MiB/s | 259.6 MiB/s |
| Warm 100 MiB read | 44.4 MiB/s | 118.6 MiB/s | 2,921.5 MiB/s |
| A1 write statements | 12,472 | 2,880 | 1,174 |
| Evidence checks | — | 99 passed | 156 passed |
| Workload | Baseline | M2 | M3 |
|---|---|---|---|
| 4 KiB random read | 2.85 ms/op | 1.17 ms/op | 0.57–1.02 ms/op |
| 100 MiB materialization | 33.8 MiB/s | 67.9 MiB/s | 108.5 MiB/s |
| 100 × 1 MiB materialization | 33.7 MiB/s | 64.6 MiB/s | 98.1 MiB/s |
| 100 one-byte edits | 8.95 s | 4.44 s | 2.13 s |
| Mixed workspace, cold / warm | 5.99 / 5.65 s | 2.28 / 2.33 s | 2.29 / 2.30 s |
| Three one-byte edits on 100 MiB | 18.6 s | 9.4 s | 70.676 ms |
| A6: 500 scattered edits | — | — | 500/500 in 9.975 s — pass |
| Workerd write-path hashing | — | 69.3 MiB/s | 383.5 MiB/s |
The 100,001-entry closure test completed with 4,655 reconciliation statements, or 0.0465 statements per manifest entry. Exact quota accounting and deduplication behavior were preserved across M2 and M3.
The smaller branch benchmark is a non-gating public-API benchmark covering 1, 5, and 10 branches, up to 1,000 changed paths, COW pages, structural patches, conflicts, replay, and limit rejection. Results below are from the local Windows x64 validation machine; preparation and publication are measured separately.
| Workload | Preparation | Publication | Outcome |
|---|---|---|---|
| 5 branches × 100 paths | 2.85 s | 0.58 s | 5 merges, 0 conflicts |
| 10 branches × 100 paths | 5.78 s | 1.23 s | 10 merges, 0 conflicts |
| 5 same-inode writers | 43 ms | 6.4 ms | 1 merge, 4 conflicts |
| 10 same-inode writers | 61 ms | 9.0 ms | 1 merge, 9 conflicts |
| 2 hard-link aliases | 12 ms | 12.9 ms | 1 merge, 1 conflict |
| 500 COW edits | 474 ms | 51 ms | 1 merge |
| 500 structural patches | 4.21 s | 5.3 ms | 1 merge |
| Replay after physical reopen | 95.6 ms | 6.9 ms | 0.26 ms replay |
The complete reduced matrix finished with 20/20 cells passing in 18.5 seconds; a three-trial conflict run passed 3/3 cells. Run it with:
pnpm bench:branchesMost branch cells use an in-memory database for repeatability; the replay cell uses a file-backed database and physical reopen. These results supplement, but do not replace, the M4 correctness and fault suites.
🧪 Benchmark methodology and caveats
- Runtime: Windows x64, Node 24.11.1, file-backed SQLite, WAL,
synchronous=FULL. - Fixtures: deterministic 100 MiB, 100 × 1 MiB, and mixed-workspace workloads.
- The matrix is a single-trial engineering benchmark with approximately 10–20% variance.
- M3's A6 acceptance gate is 500 scattered one-byte edits in ≤20 seconds. The former 1,000-edit target remains a documented SQLite WAL/fsync floor on the validation hardware, not an M3 acceptance requirement.
- Read-cell resident-memory peaks are conservative proxies because the harness currently samples the stream before consumption completes.
Full methodology and raw artifacts are in the benchmark report and M3 evidence.
Requirements: Node >=22.13 and pnpm 10.32.1.
pnpm install
pnpm validate:acceptedRun the accepted local Durable Object suite directly:
pnpm test:m6Run the M7 local selection, or the mandatory real-FUSE target on a qualifying Linux host:
pnpm test:m7:local
pnpm test:m7:fuseRun the storage engine benchmark:
node tests/performance/mini-bench.mjsRun the smaller branch-engine benchmark:
pnpm bench:branchespackages/fs/ Host-neutral filesystem core and algorithms
packages/sqlite-node/ File-backed Node SQLite adapter
packages/sqlite-cloudflare/ Cloudflare/workerd adapter
packages/node-vfs/ Node VFS integration
packages/replication/ Replication package
tests/algorithms/ M1 content and manifest tests
tests/storage/ M2 SQLite storage tests
tests/conformance/ M3 filesystem conformance tests
tests/performance/ Storage and branch benchmark harnesses and artifacts
docs/implementation/ Milestone plan and acceptance criteria
docs/evidence/ Accepted milestone evidence
docs/benchmarks/ Benchmark plans, results, and improvement targets
- M2 acceptance evidence
- M2 benchmark details
- M3 acceptance evidence
- M3 sequenced improvement plan
- M4 acceptance evidence
- M5 acceptance evidence
- M6 acceptance evidence
- M6 implementation handoff
- M7 acceptance evidence
- M7 implementation handoff
- Full implementation plan
The next milestone is M8 replication. Its Computer compatibility profile now includes authenticated fresh-replica provisioning, one shared filesystem runtime, exact active branch mounts with read-only replica main, durable resume, bounded Cap'n Web framing, generation-guarded publication, live-mount activation semantics, and an end-to-end gate through the pinned Ephemeral AI Computer fork and real FUSE. M10 remains the production cutover milestone.
LayerFS is released under the MIT License.