Guest command execution over virtio-vsock, and sandboxes that actually confine - #70
Open
admercs wants to merge 32 commits into
Open
Guest command execution over virtio-vsock, and sandboxes that actually confine#70admercs wants to merge 32 commits into
admercs wants to merge 32 commits into
Conversation
The Hypervisor trait identifies a vCPU by a bare id, so WHPX and KVM keyed `vcpu_map` by that id alone and `load_boot` resolved the partition with `vms.last()`. Two VMs on one backend therefore shared a vCPU 0, and a boot image loaded into whichever partition happened to be newest. The lookups read as "the right one" but meant "the most recent one". `VM::new` already builds a backend per VM, so the invariant held in practice and nothing was observed to be broken — but `new_with_backend` is public and hands a caller the aliasing path with no warning. Make the invariant explicit instead of implicit: `vms: Vec<_>` becomes `vm: Option<_>`, so the type says there is at most one, and `create_vm` refuses a second with an error that explains why rather than silently returning a handle onto the first. `shutdown` releases the slot, so a backend stays reusable. HVF has the same defect in a different shape: Hypervisor.framework allows one VM per process and every field is backend-global, so a second `create_vm` aliased the first outright. Guard it the same way, and split `set_up_vm` out so a failure part-way through releases the claim rather than bricking the backend. Tests: two per backend, covering the refusal and reuse-after-shutdown. Both skip on hosts without a usable hypervisor, which includes this one and CI — the wiring is verified, the runtime assertion is not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI installs `dtolnay/rust-toolchain@stable` unpinned, so it picks up 1.98.0 (2026-08-20) and its new lints on the next run. Four sites fail `-D warnings` today, independently of any branch. Three are mechanical: `Histogram::reset` and the TAP registry decode become `fill` and `as_chunks`, and `RateLimiter::check` returns `Option<u64>` — `Some(remaining)` allowed, `None` refused — instead of `Result<u64, ()>`, whose error type carried no information anyway. The fourth is not mechanical. `manual_slice_fill` fired on the two Drop impls in crypto::fips that erase key material, and `fill(0)` would have silenced it while leaving the real problem: both the loop and `fill` write to memory that is never read again, which the optimizer is free to delete entirely. Use `zeroize`, which is already in the lock file and uses volatile writes, so the erasure survives optimization. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every virtio device in this crate modelled a device that no driver could find. The queues kept their descriptor tables in host Vecs a test filled in, and nothing mapped a virtio register file into guest physical address space, so the devices were reachable from tests and from nowhere else. Three pieces, in dependency order: GuestQueue reads the descriptor table, available ring and used ring out of guest memory, so a driver in a booted guest and the device are talking about the same bytes. Every field it reads is guest-written, so each is bounded rather than trusted: a chain that cycles is refused instead of walked forever, a descriptor index past the table is an error, an indirect table may not nest, and a chain may not claim more than 64 MiB. VirtioMmioTransport is the register file, virtio-mmio version 2. It implements Device, so registering it with register_mmio_region puts it on the MMIO exit path VM::run already takes. A notify arriving before DRIVER_OK is ignored, because a queue mid-publication is not a queue. VsockDevice is the first device on that transport: the channel a guest agent will speak over, chosen because it works before the guest has networking and needs no guest address. Credit is enforced in both directions rather than assumed, and both buffers are bounded — the same shape as the serial console overflow, and the case where an agent streaming guest output would otherwise leak host memory. peek() reads without consuming, for the same reason peek_output() does. 43 tests drive these against real guest memory: the vsock tests lay out rings and publish descriptors exactly as a driver does, so the packet encoding, connection state machine and credit arithmetic are covered end to end. No test boots a kernel, so what they support is "the device implements the protocol", not "a Linux guest connected". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
execute_script evaluates a Rhai script on the host against four read-only scalars. Four places described it as running inside the guest, and its documented example was a shell command Rhai cannot parse. The engine was never the problem: there was no way to reach into a guest at all. There is now. VM::attach_vsock maps a virtio-vsock register window into guest physical address space and keeps the device, so a host process can open a connection to a program inside the guest. hv2-guest-agent holds the wire protocol -- length-prefixed JSON, one request at a time -- and the in-guest binary that answers it. GuestAgent is the host client, and AgentVM::exec_in_guest is the operation an agent calls. Four things have to be true before a command runs, and each fails as itself rather than as a generic timeout: no vsock device attached, a guest that was never told where to look, a guest that is not running, and a guest with no agent in it. "The host never attached a device" and "the guest never answered" send an operator to different places, so they do not share an error -- the same reasoning that made the console report attached separately from its contents. Running a command in a guest is gated on Capability::GuestExec, which existed and had never been consulted. Reading a VM and running arbitrary programs inside it are different powers, and VmRead should not imply the second. Deliberate limits, stated rather than implied: no streaming, output capped at 1 MiB per stream with a truncated flag, and an exit code kept distinct from a terminating signal, because a program killed by SIGKILL did not exit 0. The guest is given a shorter deadline than the host waits, so an overrun comes back as a reported timeout with whatever the program printed rather than as host-side silence. docs/GUEST_AGENT.md carries the build story: cross-compiling the binary, getting it into an image, the kernel config the vsock transport needs, and the trust model -- the channel is the boundary, and the agent runs what the host asks as whoever started it. The guest binary is Linux-only and so is not built by a Windows run; it was type-checked with --target x86_64-unknown-linux-gnu. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The mechanism landed in the previous commit had no way in from outside
the crate. This adds vm.exec to the MCP tool surface and
POST /api/v1/vms/{id}/exec beside it, so an agent and an operator with
curl reach the same guest -- the same reason the console is shared
between them.
The tool description says outright how it differs from vm.execute_script,
and a test pins that: the reason this tool exists is that another one was
described as doing what it does, and an agent picks the wrong one for the
same reason a reader did.
Every way of having no guest to run in is reported as itself. No host
installed, VM never started, no channel attached, nothing answering
inside -- four different messages, because an agent that receives a
timeout when the real problem was a missing device will retry forever,
and one that receives an empty success believes a command ran that never
did. VmHost::exec therefore defaults to refusing rather than to an empty
result: a host that tracks VMs without running them has no guest, and
fabricating one is the shape of defect execute_plan had.
vm.exec carries its own PolicyAction and its own ToolCategory rather than
borrowing ResourceRead and System. Running a program inside a guest is
not reading a resource, and it is not administering the host the VM runs
on; folding it into either would let a policy that meant to allow one
allow the other.
LocalVmHost now builds its AgentVMs holding GuestExec. Authorization for
that path lives in the MCP layer, which checks the capability and VM
ownership before dispatch -- without this the inner gate refuses what the
outer one already authorized, and the two disagree.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
IronStack renamed the layer to match the plural naming its siblings use -- Communications, Languages, Interfaces, and now Interactions. No capability, dependency or surface changed; only the layer id, which the registry and this manifest must agree on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things here were named like sandboxes and confined nothing. hv2-agent's Sandbox says so in its own docs: a policy object whose limits bind only where a caller consults them. hv2-core's container module is 3,866 lines of namespace, cgroup and seccomp structures whose start() reads "For now, simulate with a PID" and reports Running with a fabricated 1000+n. A repo-wide search for seccomp|setrlimit|unshare|prctl|CreateJobObject found prose and struct fields, and not one confinement syscall. This crate makes some, and is built around one rule: a sandbox that silently drops a control is worse than no sandbox, because a caller who asked for no network and got one believes the opposite of the truth. So controls() reports what this host enforces, determined by probing rather than by assuming; a spec asking for something the backend lacks is refused, naming the control and why it is unavailable; and best_effort() is an explicit opt-in whose result says what was dropped. Two backends behind one trait. ProcessSandbox uses cgroup v2, namespaces, rlimits and no_new_privs on Linux, and a job object on Windows. MicroVmSandbox runs the workload in a guest over the vsock agent, which is where the controls no host kernel gives an unprivileged process come from. A caller picks isolation strength without changing how it asks. Ordering is load-bearing in both, and both say why. On Linux the cgroup is joined before CLONE_NEWUSER, because afterwards the file is unwritable, and a second fork follows unshare(CLONE_NEWPID) because that flag puts the *next* child in the namespace -- without it the workload would run in the host's PID namespace while the code claimed otherwise. On Windows the child is created suspended and assigned to the job before it executes an instruction; assigning after spawn leaves a window to allocate past the cap or spawn a process that escapes the set. Deliberate refusals, rather than weaker things wearing the same name: Linux filesystem isolation is unimplemented and reported as such, because a chroot a retained descriptor walks out of is not isolation and pivot_root needs a prepared root. macOS reports resource limits and nothing else rather than building on a sandbox_init deprecated since 10.8. SandboxCommand starts with an empty environment, since inheriting would hand a sandboxed workload every credential the parent holds. Verification, unevenly: Windows is verified on this host, including a test that gives a workload a one-process job and confirms the kernel refuses the process it tries to spawn. The Linux backend is type-checked for x86_64-unknown-linux-gnu under clippy -D warnings and has never run on a kernel; the probe design means it reports what fails rather than claiming what it cannot do, but that is not the same as the isolation holding. docs/SANDBOXES.md carries the table of what is verified where. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An agent that needs to run something on the host -- a build, a linter, a conversion -- has had no way to ask, and the honest reason was that there was nothing to run it under. A "run this command" tool without confinement would have handed every agent the account the server runs as. hv2-sandbox made confinement real, so the tool can exist. sandbox.run and sandbox.capabilities dispatch against a SandboxHost, the way vm.* dispatches against a VmHost. LocalSandboxHost runs the workload under a ProcessSandbox; a deployment wanting a VM boundary instead passes a MicroVmSandbox to with_sandbox and the tools do not change. Three deliberate properties, each of which is a way this could have gone quietly wrong: With no host installed the tools refuse. The alternative to confinement is not running the program unconfined, it is not running it. The defaults are the strict ones -- 512 MiB, 30 seconds, no network, no new privileges, processes isolated -- built by relaxing SandboxSpec's untrusted spec rather than by assembling one field at a time, so a field nobody set can never mean "unconfined". Unknown fields are rejected rather than ignored: a misspelled allow_network should not silently become the default in either direction. Admin no longer implies HostExec. Every other capability is implied by the Admin wildcard, and every other tool acts on VMs the server manages; this one acts on the machine the server runs on. Leaving it in the wildcard would have granted host execution to every session already holding Admin the moment this shipped, which is a privilege expansion nobody would have written down. HostExec and PolicyAction::HostExec are separate from their GuestExec counterparts for the same reason: a program in a guest cannot reach the host and one on the host can, however well confined. The repo's own ontology invariant caught two parameters I had left without descriptions, which is what that test is for. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t found The Linux backend had only ever been type-checked. This host turns out to have a real kernel available -- WSL2 Debian, 6.18, unprivileged -- so it has now been run, and running it found two defects of the same shape: a claim that was true in the mechanism and false in what the workload could actually observe. best_effort was broken on any host without cgroup delegation. It promised to run with whatever the host could enforce, then had the backend attempt to create a cgroup the probe had already reported unavailable, and failed with ConfinementFailed. The fix is structural rather than local: Sandbox ::run now filters the spec through SandboxSpec::without_controls before handing it down, so a backend can never attempt a control its own probe rejected. /proc and /sys were inherited. Neither is an ordinary directory -- each is a view of the namespace it was mounted in -- so the workload was correctly PID 1 in its own namespace and could still enumerate 48 host processes, and had only loopback on netlink while /sys/class/net listed the host interfaces. "Cannot signal" was true; "cannot see" was not. Both namespaces now bring CLONE_NEWNS and remount the filesystem that describes them, and both probes rehearse the remount rather than only the unshare, so a kernel where it fails reports the control as unavailable. The new tests ask the workload what it can see rather than reading controls() back. That distinction is the point: a test that only checked the reported set would pass on a backend that reported everything and applied none of it. On this kernel the workload is PID 1, sees 3 processes, and has one interface. An example, `probe`, prints what any given machine can enforce and asks a confined workload what it can see. The answer differs per kernel, per container and per cgroup delegation, which is why it is a program rather than a paragraph. Linux: 20 tests, clippy -D warnings, on the kernel. Windows: 18 tests unchanged, workspace 4812 passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CI on real macOS found it: the sandbox host test asserting that a non-zero exit is a result rather than an error spawned /bin/false, which does not exist there. Every other path these tests use -- /bin/echo, /bin/sh, /bin/sleep -- is in the same place on both, so this was the only one. /bin/sh -c "exit 3" instead, which is portable and lets the assertion name a specific status rather than only "not zero". The failure was in the test, not the backend; the macOS sandbox backend itself built and its other tests passed. That job is also the first evidence the unix fallback compiles and runs on real macOS rather than only type-checking for it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous page described a boot path that had reached "able to execute a guest" and a CI run that had just gone green. Both statements have moved: there is now a channel into the guest and a sandbox that confines, and the CI picture is a pull request away from landing rather than a merged fact. Its organising device is a verification ledger -- every claim paired with the evidence behind it and where that evidence came from. That column is the point. This session started by inheriting the assumption that no Linux kernel was available here, repeated it twice, and was wrong: WSL2 Debian is one command away, and running the sandbox on it found two defects that cross-compiling could not. "Compiles" and "was run" needed to stop looking alike on the page. The previous handoff is recoverable at a7668f3 if its boot-path archaeology is wanted -- the WHPX 0x80370302 diagnosis and the CI recovery history are not repeated here. Also published as an artifact for reading outside a checkout. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This module models the OCI runtime spec: 3,866 lines of namespace, cgroup and seccomp types plus a lifecycle state machine, with nothing behind them and, as it turns out, no callers anywhere in the workspace. That would be fine. What was not fine is that it reported success. ContainerRuntime::start invented a PID -- 1000 + n -- and marked the container Running. A caller had no way to tell that from a working runtime: the state was right, the PID was plausible, and nothing was confined or even executing. It refuses now. Writing the test for that turned up the same defect in a second place. kill() returned Ok(()) for a container in any state other than Running, reporting a signal delivered to a process that did not exist. It refuses now too. Container::start is untouched and stays honest: it takes a PID from its caller, so whoever supplies one is the party that knows it is real. The tests that drove the fabricated path have been rewritten to asserts what is true -- that start refuses, that a refused start moves nothing and invents no PID, that stats count three Created containers rather than one imaginary running one -- and one now exercises the state machine directly, which is the half that works. The module header says plainly what this is and points at hv2-sandbox for confinement an operating system actually enforces. Workspace: 4,812 tests passing, clippy clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…repairs The repo keeps an Unreleased section per crate and the previous session maintained it; this session's work was missing from it entirely. Added covers the two tracks: a guest that can finally be reached, and confinement an operating system enforces. Changed carries the one entry that alters existing behaviour -- Admin no longer implying HostExec -- because that is the item most likely to surprise someone who upgrades without reading further. Fixed records the container runtime that reported success for things that never happened, and the two sandbox defects that only turned up once the code ran on a real kernel instead of cross-compiling for one. CI records three workflow steps that had never run once, and says plainly that Benchmarks stays red for a reason a one-line fix does not address. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…st double The in-guest binary had only ever been type-checked for Linux. This repo already knows that cargo check does not link -- it is why the HVF backend reached CI broken -- so "it compiles for the target" was the weakest claim in the stack. Built and run on Linux 6.18 it links to a 497 KB ELF, binds AF_VSOCK port 1024 (ss --vsock confirms the listener), accepts a connection, answers a ping, runs /bin/sh -c and returns its stdout with the program's real exit code, and refuses a mismatched protocol version with a message naming both sides. That is the framing, the read loop and the exec path exercised through the kernel rather than through the fake channel the unit tests use. The probe that did it ships as tools/vsock_probe.py, because the three things it tells apart -- no agent, wrong version, not answering -- are exactly what someone bringing up a guest image needs when vm.exec times out and the image looks fine. The docs now separate the two halves rather than calling the whole stack unverified. The agent half works. The device half still has no kernel booted against it, and the halves have never met: the agent was proven over the kernel's own vsock stack, the device against tests. Joining them is the first real boot. A local run needs vsock_loopback, which was loaded for the test and unloaded afterwards. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… was set Control::Memory was reported as enforced on Windows on the strength of SetInformationJobObject returning success. That is configuration, not enforcement -- the same gap the process-count test already closed for the other half of the job object. A workload under a 256 MiB job now asks for 1 GiB and gets System.OutOfMemoryException back from the kernel. The first version of this test asserted the workload failed, and it did not: PowerShell catches the allocation failure, reports it, and still exits 0. The exit code was never the evidence. The refused allocation is, so that is what the test reads, and the comment says why -- a later reader would otherwise "fix" it back. Windows 19 tests, Linux 20 unaffected (the test is cfg(windows)). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
platform : Kvm
provision : OK -- backend VM and vCPUs exist
launch : OK -- the guest is executing
console : "Hello, World!\r\n"
The handoff called this the gate on every boot-path claim, and said the
KVM path had never run at all -- type-checked only. It needed a machine
with an accessible /dev/kvm, which this one turns out to have: WSL2 has
nested virtualisation on, kvm_amd loaded, svm in cpuinfo. The blocker was
an inherited assumption, not hardware.
VM::provision now demonstrably creates a real KVM VM and its vCPUs,
load_boot writes a real image into guest physical memory, and launch runs
it. The 512-byte real-mode image in examples/guest_code writes to COM1 and
halts, so the evidence is the guest's own output rather than an exit code
-- and it arrived through SerialDevice, DeviceManager and
VM::console_output, which means the console plumbing from the previous
session was right.
Two probes ship with it, because the answer differs per machine and is
worth being able to ask: kvm_probe reports platform, VM::new and
provision separately, since a host can pass one and fail the next --
Windows does, when Hyper-V owns VT-x. boot_probe runs the whole path and
prints what the guest said.
Writing them turned up a detection bug. is_kvm_available checked
Path::new("/dev/kvm").exists(), which is true on any host with the module
loaded, including the very common case where the calling user is not in
the kvm group. detect() then reported Kvm and every later call failed with
a permission error naming none of this. It opens the device now, so an
unprivileged process falls back to Tcg -- the honest answer to the
question detect is actually asking. Verified both ways: as root, Kvm; as
an unprivileged user on the same machine, Tcg.
hv2-core: 2,084 tests on Linux with KVM live, 2,137 on Windows.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…startup The workload has to fit under the cap it is testing. PowerShell does not: it spends most of a 256 MiB budget on its own startup, and on a cold CI runner it spent the entire wall-clock deadline doing that, so the test failed with killed_by: WallClock and empty output -- a timeout that said nothing either way about whether the job limit binds. It passed here only because this machine is warm. The workload is now this test binary re-executed with a variable that makes it ask Windows to commit a fixed number of bytes and print which answer came back, via try_reserve_exact so a refusal is a value rather than an abort. It runs in under a second. Both directions are asserted now: refused under a cap below the request, granted under a cap above it. One-sided, the test also passes on a host that simply had no memory to spare, which is no evidence about the cap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The ledger's one genuinely unverified row said "no kernel has booted on it" and put the cause down to hardware. Half of that is now false and the other half was never true: a guest executes on KVM here, and what the vsock halves still need to meet is a guest kernel built with CONFIG_VIRTIO_VSOCKETS, not a different machine. Adds the boot and the detection fix to the changelog, adds the row to the handoff table, and rewrites the GUEST_AGENT paragraph that sent the reader looking for hardware. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Running a real bzImage against BootSource::Linux found two required pieces of the boot protocol missing. boot_params carried no e820 map. The entry count at 0x1e8 was zero and the table at 0x2d0 was empty, and a guest booted this way runs no BIOS, so there is no INT 15h to fall back on: the kernel finds no RAM at all and stops before it has a console to say so on. The map is built from the guest's memory size now. That arrives via LoadedBoot::set_memory_size rather than load(), because an API server validating an image has no VM and no size to give; asking for the memory regions without one is refused instead of answered with an empty map. No KVM vCPU had ever been given a CPUID configuration. set_cpuid and get_supported_cpuid were both written and neither was called by anything, so the guest saw a CPU reporting no vendor, no features and a maximum leaf of zero. A Linux kernel asks within its first few dozen instructions. create_vcpu applies the host's supported set now, which is what a VMM with no CPU model configured is expected to do. The visible effect is a guest that executes rather than spinning: it reaches the kernel and triple-faults, which is a failure with a next step. The kernel does not boot yet and this claims otherwise nowhere. examples/linux_boot_probe.rs reports how far it gets; the 32-bit entry state was checked separately with a hand-assembled image that executes at 1 MB and drives COM1, so what remains is specific to the Linux path. Also drops a register dump on KVM_EXIT_SHUTDOWN that looked useful and was not: on SVM, KVM resets the vCPU before returning that exit, so the registers always read as the reset vector. The comment says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An agent's history is normally kept by putting it back in the prompt. That forces the decision about what matters to be made at write time: when a tool returns 40 MB, something chooses what to keep before anyone knows what the next question will be, and whatever it discards is gone. The new crate implements the alternative from Scroll (arXiv:2608.21690). The history lives outside the context as something the agent queries, and everything rests on one invariant: eviction changes the view, never the record. EventLog is append-only and offers no operation that edits or removes an event -- not as discipline, as absent API -- and every event has a Seq that never changes and never repeats. Payloads over 8 KiB move to a store behind a handle and are still indexed by their whole content, because indexing the preview would make a large result findable only by its first 240 bytes, which is where nothing interesting ever is. WorkingView::evict persists before it selects, so nothing can leave the view before it is addressable; then it protects the active turn and the recent tail, folds unprotected tool payloads down to their addresses, and only then evicts, leaving a Headline in a tiered index. A headline is a pointer and not a summary: a summary replaces what it describes and a headline sits beside it carrying its address, so thin or wrong is survivable. The index keeps recent history detailed and coarsens distant history to O(k log_k n) blocks, and every entry at every tier still carries its span. Seven MCP tools dispatch against a ContextHost the way sandbox.* does against a SandboxHost. With none installed they refuse: a record that accepts every write and loses it is worse than none, because the agent is told it succeeded. context.exec needs HostExec as well as the new ContextMemory capability, since being able to read the record is not a reason to run code on the machine holding it. The runtime is a confined process with a durable workspace, not a resident namespace: files persist between calls and variables do not. docs/CONTEXT_AS_ENVIRONMENT.md says so, along with the log being flushed rather than fsynced and the log being unreachable from inside the sandbox by absence rather than by policy. Two rules the integration tests found rather than confirmed: the newest tool result can only be protected while it is still recent, or one old result pins the whole view and eviction silently stops working; and previews have to be bounded by the index, not only by the payload store, or a small inline payload comes back whole from every search. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three of the four are what they are: the entry state was checked and works, the context crate was checked end to end, and the resident namespace was not built. The fourth says the Linux boot protocol is still not verified, because two required pieces being present is not the same as a kernel booting. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
/bin/grep does not exist on macOS. The sandbox runs a program directly rather than through a shell, so nothing searches a PATH on our behalf and the test failed on the macOS runner with a spawn error while passing everywhere else. Same shape as the /bin/false assumption this repo hit before: ask the filesystem which of the two usual paths is there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
[ 0.020] Early memory node ranges
[ 0.021] node 0: [mem 0x0000000000001000-0x000000000009efff]
[ 0.022] node 0: [mem 0x0000000000100000-0x000000007fffffff]
Those are the two e820 entries this loader writes, read back out of the
guest. The kernel decompresses itself, enters the kernel proper, sets up
its zones and prints through SerialDevice into VM::console_output. It
does not reach userspace: it currently stops polling the DMA controller,
which is the next legacy device to bring up, and the ledger says so.
Four defects, each found by the one before it.
Single-step tracing came first, because nothing else could see the
problem. A triple fault arrives as VmExit::Shutdown and KVM resets the
vCPU on AMD before returning it, so the registers afterwards describe the
reset vector rather than the fault; a guest spinning in a tight loop
never exits at all. VM::single_step_trace records the address before each
step for exactly that reason, and keeps a bounded tail, since a guest can
run millions of instructions before it fails and the interesting part is
always the end. It located the fault in one run: 18,602 instructions in,
at a push instruction, with a stack pointer computed from boot_params.
That pointed straight at the setup header. create_boot_params copied a
fixed 0x1f1..0x250, which was the whole header under boot protocol 2.09
and has not been since; the image says where its header ends, at 0x202
plus the byte at 0x201. Everything past 0x250 reached the guest as zero,
including init_size at 0x260, which is what the kernel computes its stack
pointer from. Zero put %rsp somewhere unmapped.
With that fixed the kernel ran and immediately met three devices that
could not work behind the device manager. All three share a shape:
correct when a unit test calls them directly, broken the moment a guest
does. SerialDevice::read refused anything but one byte and the refusal
reached handle_exit as a device error that stopped the VM, so a kernel
probing the port with a word read killed its own guest; its write dropped
every byte after the first. RtcDevice and KeyboardDevice decoded absolute
ports while DeviceManager passes port - base_port, so every real access
fell through to an error arm. A 16550 and an i8042 are byte-wide register
files: a wide access walks consecutive registers now, and a port the
device does not implement reads as absent rather than as a failure.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
[ 0.850258] Run /init as init process
[ 0.851140] Kernel panic - not syncing: Attempted to kill init!
exitcode=0x00002a00
0x2a is 42, which is what the init in the initramfs returns. That line
only appears if the kernel unpacked the archive and executed the binary,
so it is the proof, and it needs no device and no privilege in the guest.
Four defects between the last commit and that line.
The initrd was placed at a fixed 32 MB. A compressed kernel unpacks into
init_size bytes from where it will run -- 62 MB from 16 MB for the kernel
tested here -- so 32 MB is inside that region for any kernel of ordinary
size, and decompression wrote over the initrd. The kernel then reported
invalid magic about bytes it had destroyed itself. It now goes as high as
it fits, under the header's initrd_addr_max, clamped to guest memory, and
is refused outright if it cannot clear the unpack region.
A FIFO reset erased the console transcript. Linux resets both FIFOs when
its 8250 driver takes over from earlyprintk, and this cleared the
transmit buffer -- so every guest that got far enough to initialise its
serial driver wiped its own boot log on the way past. On hardware that
reset discards bytes not yet sent; here a THR write is the transmission,
so there is nothing unsent and the buffer is the transcript.
Every device access was four bytes wide whatever the guest asked for: the
dispatch layer built a [u8; 4] and discarded the size the exit reported.
Register files are byte-wide and their reads have side effects, so a
one-byte outb of a character also cleared interrupt-enable, FIFO-control
and line-control. It looked like it worked, which is why it survived a
boot. The width travels with the access now.
An unhandled exception livelocked the VM. Only a double fault stopped it;
everything else was injected via a path that defaults to inject_interrupt
-- a hardware interrupt carrying the exception's vector, which does not
re-run the faulting instruction. So the guest re-took the same exception
forever: ~60,000 exits a second, state still Running, nothing saying why.
Unhandled exceptions stop the VM and name the vector.
An audit prompted by the earlier device bugs found the same shape in the
PIT, which is registered and would stop the VM on any wide read; fixed.
IdeController and VgaDevice have it too, are registered nowhere, and now
say so in their module docs rather than being changed untested.
Userspace output does not come back yet. printk polls the line-status
register, but a userspace write goes through the tty layer, which waits
for a transmit interrupt -- and with an in-kernel irqchip that has to be
injected with KVM_IRQ_LINE. The changelog and the ledger say so.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Control::FilesystemIsolation was probed as unavailable and refused, which was honest and still a hole: an agent asking for a confined filesystem got an error. It is enforced now with CLONE_NEWNS and a two-step pivot_root -- chroot was rejected for the reason the module docs already gave, that it leaves the old root mounted and a retained directory fd walks out. Ordering matters and is not visible to a compiler: the pivot has to come after the cgroup join and after uid_map is written, because those are host paths that stop existing at the pivot, and /proc and /sys have to be mounted after it or they land in the root being discarded. Two things the kernel forced. A non-recursive bind of a directory with submounts fails with EINVAL inside a user namespace, so the bind is MS_REC; but then MS_REMOUNT|MS_RDONLY covers only the top mount, leaving a submount writable inside a mount the caller was told is read-only. mount_setattr(AT_RECURSIVE) closes that, and its failure is fatal rather than ignored. Kernels before 5.12 have no mount_setattr, so the probe reports the control unavailable there rather than quietly giving them the weaker single-mount remount. Seven tests, each asking the workload what it can see rather than reading controls() back. One of them initially passed for the wrong reason: it redirected stderr to /dev/null, and an isolated root has no /dev, so both redirections failed and the refusal it asserted said nothing about the mount. The writable-direction half of the assertion caught it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t installed Two gaps closed. ResidentRuntime keeps an interpreter alive across calls behind the existing ContextRuntime trait, so a result computed once stays an object a later call can use -- the shape the paper describes, and the thing SandboxRuntime cannot do because every call is a fresh process. The trade is stated rather than hidden: a running process cannot be re-confined, so it is confined once at spawn, where SandboxRuntime confines every call and lets the spec change between them. The other gap was that nothing ever called set_context_host, so all seven context.* tools refused in any real deployment while every unit test passed -- the tools were correct and the product was useless. The server installs one now, with configuration for the record's root and the runtime's workspace, mirroring how the sandbox and VM hosts are already installed rather than inventing a second mechanism. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The release workflow has run once and all four builds failed in about seven seconds. It was not a workflow defect: every job reports steps: [] and no runner, and the check annotation says the job was not started because account payments failed or the spending limit needs raising. No amount of YAML fixes that. The defects behind it are real and would have failed the first genuine run, so they are fixed now: protoc was never installed although hv2-cli pulls in hv2-api, whose build script needs it, and the aarch64 Linux target was built through a container the host's protoc could not reach. That target is cross-compiled directly now with an explicit linker. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The resident runtime and the sandbox runtime do not supersede each other: per-call confinement or a namespace that survives, not both. The doc says which is which rather than leaving a reader to infer it, and the lockfile catches up with the crates added. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
rustdoc runs with -D warnings in CI, and an intra-doc link to a private item fails the Documentation job. The note is about a public module and the handler it names is not public, so it says so in prose instead of linking. Reproduced with RUSTDOCFLAGS="-D warnings" cargo doc --no-deps, which is what CI runs and what a plain cargo doc does not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
set_irq_line asserts and releases a line through the interrupt controller, which for KVM means KVM_IRQ_LINE on a KvmVm::irq_line that had been written and never called -- the third method in this backend found that way, after set_cpuid and set_guest_debug. Deliberately not inject_interrupt. That hands a vector straight to the vCPU and bypasses the controller's masking and priority, which is wrong whenever an in-kernel irqchip exists, and is the mistake the exception path was already making. Device::pending_interrupt lets a device report the line it is asserting, polled after every access because that is when the condition changes: a UART becomes ready to send the moment the guest writes a byte. The 16550's two sources are implemented and unit-tested. What this does NOT do is fix userspace output, and the changelog says so. What is now established: a guest reaches userspace, its init opens /dev/console, and the write returns the full byte count -- so the tty layer takes the data and never hands it to the device, and not after eight seconds either. The guest writes IER=0 every time and never sets MCR OUT2, the gate that puts a UART's interrupt on the line at all, so it is running the port without interrupts and this path is never exercised. The plumbing is a prerequisite that was missing, not a fix that was verified, and the ledger records it as delivery-unproven. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two related pieces of work, plus the CI run that neither had.
Running a command in the guest, for real
execute_scriptwas described in four places as running inside the guest andalways evaluated a Rhai script on the host. The engine was never the problem:
nothing in the repo could reach a guest at all. Virtio devices kept their
descriptor tables in host
Vecs a test filled in, and no virtio register filewas ever mapped into guest physical address space.
GuestQueue— split virtqueues read out of guest memory. Every field isguest-written, so each is bounded: a chain that cycles is refused, an index
past the table errors, indirect tables may not nest, a chain may not claim
more than 64 MiB.
VirtioMmioTransport— virtio-mmio v2. It implementsDevice, soregister_mmio_regionputs it on the MMIO exit pathVM::runalready takes.VsockDevice,hv2-guest-agent(wire protocol + in-guest binary),GuestAgent,AgentVM::exec_in_guest, surfaced asvm.execandPOST /api/v1/vms/{id}/exec.Four ways of having no guest to run in each fail as themselves rather than as a
timeout — an agent that gets a timeout when the real problem is a missing device
retries forever.
Capability::GuestExecexisted and had never been consulted.Sandboxes
Nothing in the repo confined anything.
hv2-agent'sSandboxsays so in its owndocs;
hv2-core::containeris 3,866 lines whosestart()reads "For now,simulate with a PID" and reports
Runningwith a fabricated1000 + n. Arepo-wide grep for
seccomp|setrlimit|unshare|prctl|CreateJobObjectfound proseand struct fields — not one confinement syscall.
hv2-sandboxis built around one rule: silently dropping a control is worsethan no sandbox, because a caller who asked for no network and got one believes
the opposite of the truth.
controls()reports what this host enforces, probedby attempting each one; a spec asking for what the backend lacks is refused,
naming the control and why;
best_effort()is an explicit opt-in whose resultsays what was dropped.
Two backends behind one trait:
ProcessSandbox(cgroup v2 + namespaces +rlimits +
no_new_privson Linux, job objects on Windows) andMicroVmSandboxon the vsock machinery above.
sandbox.run/sandbox.capabilitiesexpose it toagents.
Adminno longer impliesHostExec. Every other capability is implied bythe
Adminwildcard, and every other tool acts on VMs the server manages; thisone acts on the machine the server runs on. Leaving it in the wildcard would have
granted host execution to every existing admin session the moment this shipped.
What running it on a kernel found
The Linux backend was initially only type-checked. Running it on a real kernel
(WSL2 Debian 6.18, unprivileged) found two defects of the same shape — a claim
true in the mechanism and false in what the workload could observe:
best_effortwas broken on any host without cgroup delegation: itpromised to run with whatever the host could enforce, then attempted a cgroup
the probe had already reported unavailable. Fixed structurally — backends are
now handed a spec filtered to what their own probe accepted.
/procand/syswere inherited. Neither is an ordinary directory; eachis a view of the namespace it was mounted in. The workload was correctly
PID 1 in its own namespace and could still enumerate 48 host processes, and
had only loopback on netlink while
/sys/class/netlisted the host'sinterfaces. Both namespaces now remount the filesystem describing them, and
both probes rehearse the remount rather than only the
unshare.The isolation tests ask the workload what it can see rather than reading
controls()back — a test doing only the latter passes on a backend that reportseverything and applies nothing.
Verification
hv2-sandboxon Linux 6.18clippy -D warnings, on the kernelhv2-sandboxon Windows-D warningsWhat is not verified
memory and publish descriptors exactly as a driver does, so they support "the
device implements the protocol", not "a Linux guest connected". Same hardware
gate as the rest of the boot path.
--target aarch64-apple-darwin), never run.Note on history
This branch carries four commits that predate it (
9fb2db8..a7668f3) and hadnever been through CI — the main reason for opening this as a PR.
origin/masterhas since gained a manifest change (
5f67ef5) that this branch also contains asan independent commit with identical content (
319c2c9); they should mergecleanly, but it is worth knowing the duplicate is there.
🤖 Generated with Claude Code