Skip to content

learn/cow: fix the two cli_test failures CI never saw; run the full workspace suite in CI - #163

Merged
congwang-mk merged 7 commits into
mainfrom
fix-learn-cli-tests
Jul 25, 2026
Merged

learn/cow: fix the two cli_test failures CI never saw; run the full workspace suite in CI#163
congwang-mk merged 7 commits into
mainfrom
fix-learn-cli-tests

Conversation

@congwang-mk

@congwang-mk congwang-mk commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

Started as a fix for the two cli_test failures CI never saw; running the full workspace suite in CI then surfaced two deeper product bugs, both fixed here. Four fixes total plus the CI changes.

1. test_write_collapse_skips_protected: COW virtualization gap

A child under learn's workdir="/" write-opens /root/<file>. prepare_copy's confined lstat of the lower path gets EACCES (the supervisor cannot traverse 0700 /root) and treated every errno except ENOENT as an error, so the open fell through to the kernel and the child hit a real permission failure, breaking the COW promise that writes under the workdir never need real permission on the lower entry. prepare_copy now gives EACCES the same disposition execute_copy already documents (fresh empty upper file), with a 0o000-directory unit test.

2. test_learn_captures_openat2: learn granted read on "/"

The sandbox sets the child cwd to the workdir, so under learn the cwd is /; python's -c mode puts the cwd on sys.path, import ctypes lists it, and learn recorded openat("/", O_DIRECTORY) as a read grant of /, which dedup_subsumed then let swallow every other read (read = ["/"]). Reads now get the same guard the write side always had: / is never a grant, warned on stderr.

3. --uid silently degraded to 65534 on stock Ubuntu 24.04

Surfaced by the first full-suite CI run: with kernel.apparmor_restrict_unprivileged_userns=1 (the 24.04 default), unshare(CLONE_NEWUSER) yields a capability-less namespace and the child's uid_map write fails behind a let _ =, leaving the child running as the overflow uid with no indication. Mapping only runs when the caller explicitly requested an identity, so a failed map write now fails the spawn through the child's fail! path, naming the likely sysctl. CI lifts the restriction so the mapping tests exercise the real feature.

4. Supervisor deadlock: freeze racing a vfork parent

Surfaced as a CI hang and reproduced locally under CPU load (stress harness, wedged within 2 iterations). seize_and_interrupt skips tasks already in TASK_UNINTERRUPTIBLE, but the /proc state check races the tracee: a vfork parent can pass the check runnable and park in kernel_clone before the interrupt lands. The freeze then blocked in an unbounded waitpid on the supervisor loop for a task whose wait only clears when the very execve being frozen resolves. Captured live: supervisor in do_wait, vfork parent D in kernel_clone, grandchild parked in seccomp_do_user_notification.

The fix arms the interrupt first and reaps with a bounded WNOHANG loop; a task seen in D after arming becomes PendingStop (kernel-held, cannot mutate argv, traps the queued interrupt when its wait clears) and is reaped after the execve response is sent. This also narrows a small pre-existing argv-TOCTOU window: a task that passes the check runnable and then parks in D used to be skipped with nothing queued, so it could wake and run user code mid-freeze; it now carries the interrupt and traps before returning to user code. A task already in D at the pre-check is still skipped without an attachment (seizing an arbitrary uninterruptible task, e.g. one stuck on slow IO, could strand a ptrace attachment past the bounded reap), so that sliver of the window remains open. The same commit hardens the recv loop against the documented SECCOMP_IOCTL_NOTIF_RECV ENOENT race (target killed while its notification was being generated), which previously retired the whole supervisor loop.

Validation: 20/20 stress iterations clean where unfixed code wedged by iteration 2; 530 lib / 305 integration / 50 cli tests pass locally.

CI

  • Run the full workspace suite (--workspace --no-fail-fast --test-threads=1): cli, ffi, and oci suites were previously never run in CI, which is how items 1 and 2 shipped invisibly.
  • Lift the Ubuntu 24.04 AppArmor userns restriction so uid-mapping tests test the feature.
  • Bound the test step at 20 minutes so a hang fails fast with the test name visible.

Note: the prepare_copy hunk is adjacent to changes in #162; whichever merges second needs a trivial rebase, and the semantics compose (the whiteout guard sits above the lower lstat).

A child under learn's workdir="/" write-opens a file inside 0700 /root.
prepare_copy's confined lstat of the lower path fails with EACCES because
the supervisor cannot traverse the directory, and every errno except
ENOENT was treated as an error, so the open fell through to the kernel
and the child hit a real permission failure. That breaks the COW promise
that writes under the workdir never require real permission on the lower
entry. Give EACCES the same disposition execute_copy already documents
for its source open: proceed on a fresh empty upper file. Covered by a
unit test with a 0o000 lower directory.

Signed-off-by: Cong Wang <cwang@multikernel.io>
The sandbox sets the child's cwd to the workdir, so under learn the cwd
is "/". Python's -c mode puts the cwd on sys.path, so a plain import
lists it and learn recorded openat("/", O_DIRECTORY) as a read grant of
"/". dedup_subsumed then let that single grant swallow every other
observed read, producing a useless read = ["/"] profile. Apply the same
guard the write side has always had: "/" is never a grant, and the
refusal is warned on stderr so the observation is not silently dropped.

Signed-off-by: Cong Wang <cwang@multikernel.io>
CI only ran the lib and integration suites, so the cli, ffi, and oci
test binaries were never exercised. That is how two real product bugs
(the COW EACCES virtualization gap and learn granting read on "/")
shipped invisibly: the tests that catch them existed but never ran in
CI. Run cargo test across the whole workspace, with --no-fail-fast so
one failing suite does not hide the results of the others.

Signed-off-by: Cong Wang <cwang@multikernel.io>
…amespaces

Ubuntu 24.04 runner images set
kernel.apparmor_restrict_unprivileged_userns=1 by default. Under that
restriction unshare(CLONE_NEWUSER) still succeeds but yields a
capability-less namespace, the child's uid_map write fails, and the
uid-mapping tests observe the overflow uid (65534) instead of the mapped
identity. Turn the sysctl off in CI so those tests exercise the real
mapping path rather than an artifact of the runner environment.

Signed-off-by: Cong Wang <cwang@multikernel.io>
The suite normally finishes in a few minutes; the only way it runs long
is a hang, and the known failure mode is the supervisor wedging under
load. Without a bound, a wedge holds the job for the 6-hour GitHub
default and gives no hint of which test stalled. A 20-minute step
timeout turns that into a fast failure with the hanging test's name
visible in the log.

Signed-off-by: Cong Wang <cwang@multikernel.io>
The uid_map/setgroups/gid_map writes were fire-and-forget behind
"let _ =". On stock Ubuntu 24.04
(kernel.apparmor_restrict_unprivileged_userns=1) unshare(CLONE_NEWUSER)
succeeds but the map write fails, so a child that explicitly asked for
--uid silently ran as the overflow uid 65534 instead. Mapping only
happens when the caller requested a specific identity, so a failed write
is a broken contract, not a degraded mode: propagate the error and fail
the spawn through the child's fail! path, naming the likely sysctl in
the message so the failure is diagnosable.

Signed-off-by: Cong Wang <cwang@multikernel.io>
…ptible wait

seize_and_interrupt skipped tasks already in TASK_UNINTERRUPTIBLE, but
the /proc state check races the tracee: a vfork parent can pass the
check runnable and park in kernel_clone before PTRACE_INTERRUPT lands.
The freeze then sat in an unbounded waitpid on the supervisor loop for a
task whose wait clears only when the very execve being frozen resolves,
deadlocking the sandbox (seen as a CI hang, reproduced locally under CPU
load). Arm the interrupt first and reap with a bounded WNOHANG loop; a
task observed in D after arming becomes PendingStop and is reaped after
the execve response goes out, at which point its wait can clear. This
also narrows the pre-existing argv TOCTOU: a task that raced into D now
traps its queued interrupt before returning to user code instead of
running unattached. The same change stops the recv loop from retiring on
the documented SECCOMP_IOCTL_NOTIF_RECV ENOENT race (target killed while
its notification was being generated), which previously parked every
later intercepted syscall forever.

Signed-off-by: Cong Wang <cwang@multikernel.io>
@congwang-mk
congwang-mk force-pushed the fix-learn-cli-tests branch from 22d0c1a to 7a6ac56 Compare July 25, 2026 21:14
@congwang-mk
congwang-mk merged commit 6a4ec66 into main Jul 25, 2026
13 checks passed
@congwang-mk
congwang-mk deleted the fix-learn-cli-tests branch July 25, 2026 21:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant