feat(warp-core): add filesystem WAL failure atomicity

[flyingrobots]

Summary

• add host-test-only filesystem WAL fault plans for append, flush, and manifest failure injection
• let trusted runtime host tests inject filesystem WAL faults without exposing WAL authority through TrustedRuntimeApp
• prove failed pre-ACK submission append/flush leaves no witnessed submission or committed recovery evidence
• prove failed scheduler tick append/flush leaves no visible receipt/outcome and recovery remains accepted-pending with no receipt index entry
• prove manifest publish failures surface typed store errors without publishing manifest material

Closes #556

Validation

• cargo test -p warp-core --features "native_rule_bootstrap trusted_runtime host_test" --test trusted_runtime_host_loop_tests filesystem_runtime_wal_failure
• cargo test -p warp-core --features "native_rule_bootstrap trusted_runtime host_test" --test trusted_runtime_host_loop_tests runtime_wal_ack
• cargo clippy -p warp-core --features "native_rule_bootstrap trusted_runtime host_test" --test trusted_runtime_host_loop_tests -- -D warnings
• cargo check -p warp-core --features "native_rule_bootstrap trusted_runtime host_test"
• cargo fmt --check
• pnpm exec prettier --check CHANGELOG.md docs/design/reference-trusted-runtime-host-loop.md docs/design/causal-wal-hardening-matrix.md
• pnpm exec markdownlint-cli2 CHANGELOG.md docs/design/reference-trusted-runtime-host-loop.md docs/design/causal-wal-hardening-matrix.md
• scripts/check_spdx.sh CHANGELOG.md docs/design/reference-trusted-runtime-host-loop.md docs/design/causal-wal-hardening-matrix.md crates/warp-core/src/causal_wal.rs crates/warp-core/src/trusted_runtime_host.rs crates/warp-core/tests/trusted_runtime_host_loop_tests.rs
• scripts/ban-nondeterminism.sh
• git diff --check
• pre-commit and pre-push hooks

Summary by CodeRabbit

• Tests

  • Added comprehensive filesystem WAL fault injection tests for append, flush, and manifest operations to validate submission and tick rollback scenarios.

• Documentation

  • Updated design specifications with explicit filesystem fault-failure requirements and test coverage.
  • Clarified that filesystem WAL fault injection is restricted to host tests only.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 6df61cfe-49ba-477e-8542-0f34cbc0e9c9

📥 Commits

Reviewing files that changed from the base of the PR and between 7201f53 and 3048840.

📒 Files selected for processing (6)

• CHANGELOG.md
• crates/warp-core/src/causal_wal.rs
• crates/warp-core/src/trusted_runtime_host.rs
• crates/warp-core/tests/trusted_runtime_host_loop_tests.rs
• docs/design/causal-wal-hardening-matrix.md
• docs/design/reference-trusted-runtime-host-loop.md

---

📝 Walkthrough

Walkthrough

Adds cfg-gated (test/host_test) filesystem WAL fault injection to FilesystemWalStore via new FilesystemWalFaultTarget and FilesystemWalFaultPlan types. Pre-checks in append_frame, flush_commit, and publish_manifest return injected WalStoreError::Io when a plan is active. A delegation chain through TrustedRuntimeWal and TrustedRuntimeWalStore surfaces fault injection on TrustedRuntimeHost. Five new integration tests assert rollback behavior and read-only recovery state. Design docs and changelog updated.

Changes

Filesystem WAL Fault Injection

Layer / File(s)	Summary
Fault injection types and FilesystemWalStore wiring crates/warp-core/src/causal_wal.rs	Defines FilesystemWalFaultTarget and FilesystemWalFaultPlan with per-target failure counters. Adds fault_plan field to FilesystemWalStore. Inserts cfg-gated pre-checks in append_frame, flush_commit, and publish_manifest that decrement counters and short-circuit with WalStoreError::Io when a fault is configured. Adds open_with_fault_plan_for_test and replace_fault_plan_for_test APIs.
TrustedRuntimeHost fault-plan propagation crates/warp-core/src/trusted_runtime_host.rs	Extends TrustedRuntimeWalStoreConfig::Filesystem with fault_plan: Option<FilesystemWalFaultPlan>. Adds filesystem_with_fault_plan_for_test constructor on TrustedRuntimeWalConfig. Conditionally opens the store with a fault plan in TrustedRuntimeWalStore::open. Adds replace_filesystem_fault_plan_for_test through TrustedRuntimeWal → TrustedRuntimeWalStore → FilesystemWalStore, exposed as inject_runtime_wal_filesystem_fault_for_test on TrustedRuntimeHost. In-memory stores return WalStoreError::Io.
Fault injection integration tests and helpers crates/warp-core/tests/trusted_runtime_host_loop_tests.rs	Adds deterministic Blake3-based WriterEpochRequest test helpers. Adds five #[test] cases: submission append-frame failure, submission flush-commit failure, tick append-frame failure, tick flush-commit failure, and manifest publish failure. Each verifies injected error content, in-memory runtime state (witness count, intent outcome), and read-only recovery (committed transaction counts, receipt index absence).
Hardening matrix and reference docs docs/design/causal-wal-hardening-matrix.md, docs/design/reference-trusted-runtime-host-loop.md, CHANGELOG.md	Expands Slice 89 and Slice 91 requirements with concrete filesystem append/flush/manifest failure rollback assertions. Adds a clarifying sentence that fault injection is host-test-only and not exposed through TrustedRuntimeApp. Changelog updated.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

• #556 [GP1-S3] Filesystem Runtime WAL Failure Atomicity — This PR directly implements the acceptance criteria: injects append/flush/manifest failures around submission and tick boundaries, proves rollback via test assertions on witness state and read-only recovery, and provides the cargo test invocation specified in the issue scope.
• #555 — Fault injection types and test-only APIs added here build on the FilesystemWalStore and TrustedRuntimeWalConfig infrastructure established in GP1-S2.
• #554 — The PR preserves the TrustedRuntimeApp authority boundary explicitly documented in this issue by restricting all fault injection paths to cfg(any(test, feature = "host_test")).

Suggested labels

tooling

Poem

Append fails, flush cries, the manifest stays bare,
The WAL rolls back — no phantom commits there.
Counters decrement, errors surface clean,
Recovery scans and finds the slate pristine.
🔩 Fault injected, proof complete, the rollback holds the line.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat(warp-core): add filesystem WAL failure atomicity' directly summarizes the main change: adding filesystem WAL failure atomicity mechanisms.
Linked Issues check	✅ Passed	PR implements all coding requirements from #556: filesystem WAL fault injection for append/flush/manifest failures, test proofs for pre-ACK rollback and tick failure atomicity, and recovery state validation.
Out of Scope Changes check	✅ Passed	All changes are scoped to filesystem WAL failure atomicity: fault injection infrastructure, host-test-only plumbing, comprehensive tests, CHANGELOG, and design documentation updates.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch cycle/556-filesystem-runtime-wal-failure-atomicity

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}