learn/cow: fix the two cli_test failures CI never saw; run the full workspace suite in CI - #163
Merged
Conversation
A child under learn's workdir="/" write-opens a file inside 0700 /root. prepare_copy's confined lstat of the lower path fails with EACCES because the supervisor cannot traverse the directory, and every errno except ENOENT was treated as an error, so the open fell through to the kernel and the child hit a real permission failure. That breaks the COW promise that writes under the workdir never require real permission on the lower entry. Give EACCES the same disposition execute_copy already documents for its source open: proceed on a fresh empty upper file. Covered by a unit test with a 0o000 lower directory. Signed-off-by: Cong Wang <cwang@multikernel.io>
The sandbox sets the child's cwd to the workdir, so under learn the cwd
is "/". Python's -c mode puts the cwd on sys.path, so a plain import
lists it and learn recorded openat("/", O_DIRECTORY) as a read grant of
"/". dedup_subsumed then let that single grant swallow every other
observed read, producing a useless read = ["/"] profile. Apply the same
guard the write side has always had: "/" is never a grant, and the
refusal is warned on stderr so the observation is not silently dropped.
Signed-off-by: Cong Wang <cwang@multikernel.io>
CI only ran the lib and integration suites, so the cli, ffi, and oci test binaries were never exercised. That is how two real product bugs (the COW EACCES virtualization gap and learn granting read on "/") shipped invisibly: the tests that catch them existed but never ran in CI. Run cargo test across the whole workspace, with --no-fail-fast so one failing suite does not hide the results of the others. Signed-off-by: Cong Wang <cwang@multikernel.io>
…amespaces Ubuntu 24.04 runner images set kernel.apparmor_restrict_unprivileged_userns=1 by default. Under that restriction unshare(CLONE_NEWUSER) still succeeds but yields a capability-less namespace, the child's uid_map write fails, and the uid-mapping tests observe the overflow uid (65534) instead of the mapped identity. Turn the sysctl off in CI so those tests exercise the real mapping path rather than an artifact of the runner environment. Signed-off-by: Cong Wang <cwang@multikernel.io>
The suite normally finishes in a few minutes; the only way it runs long is a hang, and the known failure mode is the supervisor wedging under load. Without a bound, a wedge holds the job for the 6-hour GitHub default and gives no hint of which test stalled. A 20-minute step timeout turns that into a fast failure with the hanging test's name visible in the log. Signed-off-by: Cong Wang <cwang@multikernel.io>
The uid_map/setgroups/gid_map writes were fire-and-forget behind "let _ =". On stock Ubuntu 24.04 (kernel.apparmor_restrict_unprivileged_userns=1) unshare(CLONE_NEWUSER) succeeds but the map write fails, so a child that explicitly asked for --uid silently ran as the overflow uid 65534 instead. Mapping only happens when the caller requested a specific identity, so a failed write is a broken contract, not a degraded mode: propagate the error and fail the spawn through the child's fail! path, naming the likely sysctl in the message so the failure is diagnosable. Signed-off-by: Cong Wang <cwang@multikernel.io>
…ptible wait seize_and_interrupt skipped tasks already in TASK_UNINTERRUPTIBLE, but the /proc state check races the tracee: a vfork parent can pass the check runnable and park in kernel_clone before PTRACE_INTERRUPT lands. The freeze then sat in an unbounded waitpid on the supervisor loop for a task whose wait clears only when the very execve being frozen resolves, deadlocking the sandbox (seen as a CI hang, reproduced locally under CPU load). Arm the interrupt first and reap with a bounded WNOHANG loop; a task observed in D after arming becomes PendingStop and is reaped after the execve response goes out, at which point its wait can clear. This also narrows the pre-existing argv TOCTOU: a task that raced into D now traps its queued interrupt before returning to user code instead of running unattached. The same change stops the recv loop from retiring on the documented SECCOMP_IOCTL_NOTIF_RECV ENOENT race (target killed while its notification was being generated), which previously parked every later intercepted syscall forever. Signed-off-by: Cong Wang <cwang@multikernel.io>
congwang-mk
force-pushed
the
fix-learn-cli-tests
branch
from
July 25, 2026 21:14
22d0c1a to
7a6ac56
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Started as a fix for the two cli_test failures CI never saw; running the full workspace suite in CI then surfaced two deeper product bugs, both fixed here. Four fixes total plus the CI changes.
1. test_write_collapse_skips_protected: COW virtualization gap
A child under learn's
workdir="/"write-opens/root/<file>.prepare_copy's confined lstat of the lower path gets EACCES (the supervisor cannot traverse 0700/root) and treated every errno except ENOENT as an error, so the open fell through to the kernel and the child hit a real permission failure, breaking the COW promise that writes under the workdir never need real permission on the lower entry.prepare_copynow gives EACCES the same dispositionexecute_copyalready documents (fresh empty upper file), with a 0o000-directory unit test.2. test_learn_captures_openat2: learn granted read on "/"
The sandbox sets the child cwd to the workdir, so under learn the cwd is
/; python's-cmode puts the cwd onsys.path,import ctypeslists it, and learn recordedopenat("/", O_DIRECTORY)as a read grant of/, whichdedup_subsumedthen let swallow every other read (read = ["/"]). Reads now get the same guard the write side always had:/is never a grant, warned on stderr.3. --uid silently degraded to 65534 on stock Ubuntu 24.04
Surfaced by the first full-suite CI run: with
kernel.apparmor_restrict_unprivileged_userns=1(the 24.04 default), unshare(CLONE_NEWUSER) yields a capability-less namespace and the child'suid_mapwrite fails behind alet _ =, leaving the child running as the overflow uid with no indication. Mapping only runs when the caller explicitly requested an identity, so a failed map write now fails the spawn through the child'sfail!path, naming the likely sysctl. CI lifts the restriction so the mapping tests exercise the real feature.4. Supervisor deadlock: freeze racing a vfork parent
Surfaced as a CI hang and reproduced locally under CPU load (stress harness, wedged within 2 iterations).
seize_and_interruptskips tasks already inTASK_UNINTERRUPTIBLE, but the/procstate check races the tracee: a vfork parent can pass the check runnable and park inkernel_clonebefore the interrupt lands. The freeze then blocked in an unboundedwaitpidon the supervisor loop for a task whose wait only clears when the very execve being frozen resolves. Captured live: supervisor indo_wait, vfork parentDinkernel_clone, grandchild parked inseccomp_do_user_notification.The fix arms the interrupt first and reaps with a bounded
WNOHANGloop; a task seen inDafter arming becomesPendingStop(kernel-held, cannot mutate argv, traps the queued interrupt when its wait clears) and is reaped after the execve response is sent. This also narrows a small pre-existing argv-TOCTOU window: a task that passes the check runnable and then parks inDused to be skipped with nothing queued, so it could wake and run user code mid-freeze; it now carries the interrupt and traps before returning to user code. A task already inDat the pre-check is still skipped without an attachment (seizing an arbitrary uninterruptible task, e.g. one stuck on slow IO, could strand a ptrace attachment past the bounded reap), so that sliver of the window remains open. The same commit hardens the recv loop against the documentedSECCOMP_IOCTL_NOTIF_RECVENOENT race (target killed while its notification was being generated), which previously retired the whole supervisor loop.Validation: 20/20 stress iterations clean where unfixed code wedged by iteration 2; 530 lib / 305 integration / 50 cli tests pass locally.
CI
--workspace --no-fail-fast --test-threads=1): cli, ffi, and oci suites were previously never run in CI, which is how items 1 and 2 shipped invisibly.Note: the
prepare_copyhunk is adjacent to changes in #162; whichever merges second needs a trivial rebase, and the semantics compose (the whiteout guard sits above the lower lstat).