cow: subtree-aware whiteouts with a durable deletion log and type-aware copy-up#162
cow: subtree-aware whiteouts with a durable deletion log and type-aware copy-up#162congwang-mk wants to merge 13 commits into
Conversation
|
Thanks for taking on the whole deletion model — the subtree-aware whiteouts and the merged-view I re-ran each of the four as an in-cage test against this branch. A few things I would want resolved before it lands. Blocker — #158 is only half-closed; the hang is relocated to
Repro: lower FIFO Fix options: Should-fix — a directory rename onto an existing lower-only destination silently merges the two trees (new, reachable via #160).
Should-fix — #159 gap: the
Should-fix —
Nice-to-have —
Coupling with #148 — worth deciding the sequencing before either merges. These two rework the same
Suggested order: land Only the #158 commit hang is a hard blocker for me; the rest is should-fix/nice-to-have plus the sequencing call. |
|
Pushed the repros so you can run them directly: Three fail here — Two notes on how they are written: the FIFO one drives |
c712baf to
5f61e77
Compare
The branch tracked deletions as a flat in-RAM set of exact paths, which is the root of the #159/#160/#161 family: deleting a directory hid only the directory node itself, children stayed visible through the lower layer, and nothing about a deletion survived the process. Introduce a DeletionSet whose covers() hides a path and everything beneath it, matching per path component the way removing a directory hides its contents, and mirror every insertion to an append-only escaped log beside the upper so the deletion half of a change set is as self-describing on disk as the upper directory already is. Entries are never removed: a re-created path is exposed by upper presence shadowing the whiteout, which keeps the set append-only and the log a pure record. A log write failure degrades durability, not correctness, so it is deliberately not fatal. This commit only adds the type; the branch adopts it next. Signed-off-by: Cong Wang <cwang@multikernel.io>
…visibility and resurrection (#159) The flat deleted set answered only exact-path questions, so after the child removed a lower directory its children still resolved through the lower layer: reads and stats leaked pre-delete content, listings showed ghost entries, and an O_CREAT over a deleted file copied the old bytes back up as if the delete never happened (#159). Adopt DeletionSet everywhere the branch consults deletions: is_deleted now means covered and not shadowed, prepare_copy treats a covered path as having nothing to copy so creates over a whiteout start fresh, list_merged_dir filters covered children, changes() reports a re-created entry as a single Add rather than Delete plus Modify, and commit applies deletions before overlaying the upper, which is exactly the opaque-directory merge order. Re-creating an entry in the upper shadows the whiteout instead of erasing it, so a deleted-then-recreated directory stays opaque and the set stays append-only. Signed-off-by: Cong Wang <cwang@multikernel.io>
rmdir went straight to the path-based whiteout, so a directory that still had children visible to the child was deleted while reporting success, and the whole subtree the child never emptied vanished at commit (#161). A real rmdir refuses with ENOTEMPTY, and the branch has to give the same answer from the merged view because neither layer alone can: entries may live only in the upper, only in the lower, or be hidden by whiteouts. Answer emptiness from list_merged_dir and refuse with ENOTEMPTY while any child is still visible, so rmdir only ever deletes what the child actually drained. Signed-off-by: Cong Wang <cwang@multikernel.io>
handle_rename staged only the single source entry, so renaming a lower directory moved an empty upper husk: the merged view showed the new name without its children, and commit published the rename as data loss (#160). Stage the source tree into the upper first, entry by entry through the same confined, quota-accounted single-entry copy-up so symlinks are copied verbatim and never followed out of the tree (issue #112), and skip children already hidden by whiteouts so deleted entries do not reappear under the new name. Only then rename within the upper and whiteout the surviving lower source. Signed-off-by: Cong Wang <cwang@multikernel.io>
…158) prepare_copy assumed every lower entry holds copyable bytes and opened it to copy, but opening a FIFO blocks until the other end appears, so a write to a lower FIFO under a COW workdir hung the supervisor at copy-up time (#158). Classify the lower entry from the confined lstat and dispatch on the type: symlinks are recreated verbatim, directories are created empty, and non-regular entries (FIFOs, sockets, device nodes) take a Passthrough plan that defers to the kernel instead of reading content that is not filesystem data. execute_copy re-checks the source type after opening so a swap between classification and copy cannot make it read a special file either. Signed-off-by: Cong Wang <cwang@multikernel.io>
…sions The deletion model has two halves that must keep agreeing: what the child sees mid-run (the merged view) and what lands on disk (commit applying deletions before overlaying the upper). Every bug in the #159/#160/#161 family was a disagreement between them, and each half can regress independently of the other. Pin the agreement with branch unit tests and with end-to-end regressions that drive rm, rmdir and mv through a real sandboxed child, so a future change to either half fails loudly instead of silently drifting apart. Signed-off-by: Cong Wang <cwang@multikernel.io>
…d of passing through (#158) The Passthrough plan sent writes to non-regular lower entries back to the kernel, which re-runs the original syscall against the real file: the child then needed real write permission on the lower entry and its write really reached it. Both break the COW promise that writes under the workdir neither require permission on nor cause side effects through the real filesystem; learn mode COWs whole trees, so even `> /dev/null` landed here needing a Landlock grant, and an O_WRONLY open of a reader-less FIFO merely relocated the #158 hang from the supervisor into the child. Drop Passthrough and virtualize the write like any other COW write, onto an empty regular stub created in the upper without ever opening the source. Signed-off-by: Cong Wang <cwang@multikernel.io>
handle_rename staged the source into the upper and renamed it there without ever looking at the destination. A directory renamed onto an existing lower-only directory therefore committed the union of the two trees while reporting success, a state no real filesystem transition produces. The destination check has to come from the merged view, since the kernel only ever sees one layer. Classify both endpoints in the merged view before staging: a non-empty directory destination refuses with ENOTEMPTY, type mismatches refuse with EISDIR/ENOTDIR, and a destination that survives in the lower layer is whiteouted after the upper rename so commit replaces it instead of merging into it. handle_rename now answers the child directly with an errno, the handle_unlink convention, which also stops a whiteouted source from falling through to a real host rename in chroot mode. Signed-off-by: Cong Wang <cwang@multikernel.io>
Both open handlers split on O_DIRECTORY before consulting the deletion set, so opendir on a whiteouted directory skipped to the kernel and succeeded on the surviving lower directory. The child then held a live fd whose fstat leaked the deleted directory's inode metadata while stat on the same path answered ENOENT, breaking the present-as-absent invariant the merged view pins everywhere else. Check is_deleted before the O_DIRECTORY early-return in handle_open and prepare_open so the open answers ENOENT like every other path to a deleted entry. Signed-off-by: Cong Wang <cwang@multikernel.io>
copy_up_tree swallowed stat and read_dir failures, so a mid-tree EACCES or EIO truncated the staged copy silently; handle_rename then whiteouted the source anyway and the untraversed children were gone at commit while the child saw success. It also recursed over child-controlled directory depth, letting a pre-seeded deep lower tree grow the supervisor's stack without bound, and a mid-tree quota failure left the reservation charged and an orphaned partial subtree in the upper. Propagate every staging error to the child as its errno, walk the tree with an explicit worklist instead of recursion, and on failure remove the upper entries the staging created and re-derive the quota, so a failed rename leaves the branch exactly as it was. Signed-off-by: Cong Wang <cwang@multikernel.io>
…ntry The log escaped backslash and newline but not carriage return, while replay reads lines through BufRead::lines, which strips a trailing \r. A path ending in \r therefore round-tripped to its \r-less sibling: covers() would hide the wrong entry and the actually-deleted path would resurrect. Escape \r the same way as \n. Durability had a matching gap at the directory level: insert fdatasyncs the log file, but nothing ever synced the branch directory, so a power loss after the first deletion could drop the log's directory entry entirely, the exact crash class the log exists for. Sync the parent directory once when the log is created or reopened. Signed-off-by: Cong Wang <cwang@multikernel.io>
handle_cow_stat is registered for the legacy x86_64 variants but read every notification with the at-variant layout, dirfd in args[0] and path in args[1]. Legacy stat carries the path in args[0], so the handler read the statbuf pointer as the path, never matched the workdir, and fell through to the kernel: a static-libc child emitting legacy stat saw whiteouted lower entries that newfstatat already reported as ENOENT, and legacy lstat picked its follow behavior from a junk flags register. Same register-layout bug handle_cow_open already fixed for legacy open. Select path, statbuf and follow semantics by syscall number, and route legacy access to the faccessat existence check instead of the stat writeback, which would have written a stat struct through access's mode argument. Signed-off-by: Cong Wang <cwang@multikernel.io>
5f61e77 to
88b85d5
Compare
Mutation-testing the rename destination logic showed the whiteout of a surviving lower destination was unprotected: with mark_deleted(new_rel) deleted, every rename test still passed, because an empty-directory or regular-file destination commits identical bytes whether the rename replaces the destination or merges into it. The line only becomes observable with a special-file destination, and the failure it prevents there is severe: a FIFO destination hangs commit's publish open, the same unrecoverable class as the #158 commit hang. Renaming onto a lower symlink exercises the same code path but fails cleanly instead of hanging: without the whiteout the symlink survives into commit, whose O_NOFOLLOW publish refuses to write through it. Verified the test fails with the whiteout removed and passes with it. Signed-off-by: Cong Wang <cwang@multikernel.io>
|
Thanks for the reproducer and review, @dzerik . I have addressed them. Please take another look. |
|
Thanks — I re-ran the repros against Closed on my side: the The #158 commit-time hang still reproduces, though:
Copy-up time is genuinely fixed; it is only the publish path that is left. Whiteouting the lower path when the stub is created (so commit unlinks-then-creates), unlinking a type-mismatched destination before opening it, or opening the destination |
Redesign of the seccomp COW branch's deletion model. The flat exact-match
HashSet<String>(inherited from the Python-eracowfs/_branch.py, carried verbatim into the Rust rewrite) is replaced by a subtree-aware whiteout set, and copy-up becomes file-type-aware.Fixes #158
Fixes #159
Fixes #160
Fixes #161
The model
A new
DeletionSet(cow/deletions.rs) owns whiteout semantics:covers(rel)matches per path component: a whiteout ondhidesd/xandd/x/y, neverd2.rmdir d; mkdir dan opaque directory for free: the newdis visible, its old contents stay hidden.deleted.logbesideupper/and fdatasynced, so the deletion half of a change set is now self-describing on disk, like the upper already was. RAM stays the in-process source of truth; the log is the durability record a future recovery sweep (RFC RFC: Multi-sandbox transactions for pipelines #65 Phase 3, PR pipeline: transactional sequential pipelines over a shared COW workdir (RFC #65 Phase 1) #148's preserved branches) can replay.All merged-view queries (
is_deleted, open/stat/readlink,list_merged_dir,changes(),commit()) route through it, instead of each handler re-implementing its own slice of overlay semantics.Per issue
commit()to republish. Also fixes a latent bug of the same family found during the work:O_CREATon a whiteouted file used to resurrect the pre-delete lower bytes.> /dev/nullinside a learn-mode tree needs no real permission on/devand never touches the device) and removes the supervisor wedge.execute_copyadditionally guards withO_NONBLOCK+ fstat against the entry changing type between the classifying lstat and the open.The invariant
One test pins the property all four issues violated: the merged view the child observed during the run is exactly what
commit()publishes. Plus each issue's own reproducer as a unit test, and three end-to-end regressions driving a real sandboxed child (rm -r,mv dir,rmdirnon-empty).Compatibility
No public API, CLI, or SDK signature changes. Child-visible behavior changes only toward kernel semantics (ENOTEMPTY, ENOENT under deleted dirs, dir renames preserving contents). The branch storage dir gains
deleted.log; the layout is internal.Suites: 555 lib / 308 integration / 386 Python locally; the two
cli_testfailures on my machine (test_learn_captures_openat2,test_write_collapse_skips_protected) reproduce identically on main and are unrelated.