Skip to content

Deforkify event loop - #2

Merged
danlapid merged 12 commits into
danlapid:mainfrom
guybedford:deforkify-event-loop
Sep 26, 2026
Merged

Deforkify event loop#2
danlapid merged 12 commits into
danlapid:mainfrom
guybedford:deforkify-event-loop

Conversation

@guybedford

@guybedford guybedford commented Sep 17, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Follow-up to #1: the server moves from JSPI onto Tokio's host-driven event loop, and onto the released tooling around it. The Worker is written against the worker crate and built by worker-build --emscripten, runs on stock Wrangler/workerd, and the world is a mounted SQLite filesystem again.

Changed vs #1:

  • runtime: connect is a #[wasm_bindgen(experimental_tokio)] export on Tokio's LocalEventLoop (rt: support hosted event loops on wasm32-unknown-emscripten tokio-rs/tokio#8484); the host event loop is the runtime's wait, exports return promises and nothing blocks or suspends. No -sREENTRANT_JSPI, no shadow-stack switching
  • build: the worker crate (git dependency on worker-build --emscripten cloudflare/workers-rs#1061, experimental_tokio feature) and worker-build --emscripten, installed by setup from the same revision; worker-build provisions Emscripten (emsdk 6.0.10 + its bundled epoll-listener/async-DNS patches) and the wasm-bindgen CLI itself, so .work/ holds only the patched Pumpkin checkout. Replaces the hand-written JS entrypoint and the direct emcc/wasm-bindgen invocation
  • workerd: stock Wrangler 4.141.0 / workerd 1.20260925.1; the object is declared under exports (the deployed Worker's existing configuration) and reached through cloudflare:workers' exports. node:net: add net.Server and cloudflare:node connectHandler cloudflare/workerd#7306 and #7313 (net.Server routing, handleAsNodeConnection) and the node:fs fixes #7368, #7369, #7393, #7394 have all shipped, so MINIFLARE_WORKERD_PATH is gone
  • persistence: the world is a live SQLite-backed filesystem, mounted at /data through worker-fs-mount with the durable-object-fs backend, so every Pumpkin write is durable as it happens; the tree-to-SQLite checkpoint/restore (persist.rs, the files table) is removed and status reports saved_at
  • Pumpkin patch: single-threaded builds run Rayon, chunk-generation and blocking jobs as Tokio tasks (one turn of the loop each) instead of inline; StopSignal/reset_stop let an object start another server after one stops
  • stdio: the console override is removed; Pumpkin's stdout/stderr go through NODERAWFS to workerd's descriptors and Wrangler relays them as stdout:/stderr: lines

Also brings in dlapid/moreMemoryOpts (ec02ab2, cd83251), ported onto this runtime so everything lands together:

  • Pumpkin memory work: packed chunk palettes with copy-on-write light arrays, on-demand carving masks, bounded density pools, pruned structure caches, lazily grown pathfinder heaps; byte-bounded outgoing chunk batches with backpressure and idle generation-graph release. Two players now sit at ~66-74 MB of Wasm memory (was ~82 MB)
  • operator settings as Worker variables (VIEW_DISTANCE, SIMULATION_DISTANCE, MAX_PLAYERS, COMPRESSION_THRESHOLD/LEVEL, MOTD, WORLD_SEED, IDLE_TIMEOUT_SECONDS), validated in the object and reported in status; packet compression on by default. See docs/configuration.md
  • the Worker's #[event(connect)] handler reads the Minecraft handshake over the worker::Socket: server-list pings are answered from object metadata without starting a server, and login connections are forwarded with Stub::connect and copied with Tokio; the object is a #[durable_object(connect)] whose handler hands each socket to Pumpkin's listener with Socket::handle_as_node_connection and awaits it (the mount uses state.storage().as_raw(), the final save Storage::sync)
  • after the last player leaves the server saves and keeps a reconnect window before stopping (idle_deadline, runtime_starts in status)
  • compression-aware test client, unit tests for framing, an experience probe, and integration coverage for pings/limit/reconnect/restart
  • not taken from that branch: the injected-stream Pumpkin entry point, the chunk-writer checkpoint barrier and the TypeScript Worker (superseded by the stock accept loop, the mounted filesystem and the Rust Worker), plus the ring/tokio/proc-macro-error2 build patches that the pinned toolchain no longer needs

worker-fs-mount 0.2.0 rejects the S_IFREG | perms mode Emscripten's FS.open passes to its backend (Node masks numeric modes to 0o7777); the fix is danlapid/worker-fs-mount#10, and until it ships npm install applies it to the installed package as patches/worker-fs-mount-open-mode.patch.

PR branches referenced directly, now as tags

To run - same instructions as the README, on stock Wrangler; no workerd, Emscripten or wasm-bindgen build needed. bash scripts/setup.sh provisions everything (Pumpkin checkout, worker-build, npm ci); npm test passes end to end from a clean checkout, printing PUMPKIN-DO-SQLITE-RESTART-OK after verifying player and chunk restoration across a restart.

@guybedford
Use released socket2 and wasm-streams rlib-only branch
82c81ad 
Drop the wasm-streams checkout and cdylib patch in favor of
guybedford/wasm-streams (MattiasBuelens/wasm-streams PR pending). The
socket2 git rev was exactly the v0.6.5 release.
@guybedford
Run Pumpkin bottom-up on stock Tokio under JSPI
4cb2734 
One perpetual `#[wasm_bindgen(jspi)]` export builds a current-thread
runtime and `block_on`s the whole server lifetime; every park suspends the
Wasm stack on `epoll_wait`, so the hosted runtime adapter is gone. Pumpkin
binds its stock `TcpListener` on 25565 inside the Durable Object's port
table, and the object routes each inbound socket to it with
`handleAsNodeConnection`, replacing the injected-stream entry point,
wasm-streams, and the workers-rs dependency. `stop` cancels the server and
the run promise settling is the checkpoint signal. `-sREENTRANT_JSPI`
gives each activation its own shadow stack, so other entries into the
module while the server is suspended cannot clobber its frames.

Toolchain: Rust beta; emscripten main plus the JSPI hooks, reentrant JSPI
and epoll listener PRs, with the paired emscripten-releases LLVM and the
jspi-hooks Binaryen branch built by setup; wasm-bindgen 0.2.128 CLI via
`-sWASM_BINDGEN`; exnref exception handling throughout; tokio
`emscripten-epoll`, mio tokio-rs/mio#1969, libc `libc-0.2`. rustc needs a
larger compile-thread stack for pumpkin-data. The wasm-bindgen and
workers-rs patches and the CLI lockfile are gone; the Pumpkin patch drops
the injected-stream entry point.

Requires a workerd with per-Durable-Object port tables and `net.Server`
inbound routing (`MINIFLARE_WORKERD_PATH`). CI moves to Linux.
@guybedford
Run the Durable Object on Tokio's EventLoopRuntime
caaa63b 
The whole Worker is now Rust linked directly by cargo and emcc, with no
JSPI and no JS-side driver. The Durable Object owns an EventLoopRuntime
(tokio-rs/tokio#8479): Tokio's current-thread scheduler and drivers wait on
the host event loop through Emscripten's epoll listeners
(emscripten-core/emscripten#27547). Exports return promises; the server
lifetime is a scheduled root that completes by callback after the final
save and checkpoint. Connections arriving while the server starts or stops
wait for the next phase change and retry.

worker/index.mjs re-exports the generated module and derives the object
class from DurableObject for RPC. The worker and worker-build dependencies,
the TypeScript worker, and the filesystem fixtures are removed.

The Pumpkin patch makes the stop token replaceable behind a StopSignal and
adds reset_stop so a server can start again in the same instance. Under
NODERAWFS the host enforces permissions; src/workerd.js forces
FS.ignorePermissions since workerd reports directories without execute
bits, and keeps stdio on the console callbacks.

Documentation describes the event-loop model and the workerd node:fs
requirements (positional buffer I/O, O_TRUNC, O_CREAT, rename over an
existing path).
@guybedford
guybedford force-pushed the deforkify-event-loop branch 4 times, most recently from b48bd74 to bae8d13 Compare September 24, 2026 06:39
@guybedford
guybedford marked this pull request as ready for review September 24, 2026 06:49
@guybedford
guybedford force-pushed the deforkify-event-loop branch 7 times, most recently from dfe2fbe to 16a7895 Compare September 25, 2026 21:25
@guybedford
Build with the worker crate and worker-build --emscripten
3ef2892 
The Worker is now written against the `worker` crate with its
`experimental_tokio` feature, built by worker-build's Emscripten mode:
worker-build owns the codegen and link settings, wraps the exports into the
entrypoint and derives the Durable Object class for RPC, so the hand-written
JavaScript entrypoint goes away. `connect` is a
`#[wasm_bindgen(experimental_tokio)]` export on Tokio's host-driven event
loop.

The toolchain follows the sealed releases: worker-build (cloudflare/workers-rs
#1061, installed by setup from the revision Cargo.toml pins the `worker` crate
to) provisions Emscripten 6.0.10 with the epoll-listener and async-DNS patches
(tag 6.0.10-cf.emscripten) and the wasm-bindgen 0.2.129 CLI itself; Tokio
1.53.1 with the event loop (tag 1.53.1-cf.emscripten) and mio 1.2.3 with the
Emscripten selector (tag 1.2.3-cf.emscripten) are git dependencies, and
wasm-bindgen and wasm-streams 0.7 come from crates.io. Setup keeps only the
patched Pumpkin checkout under .work; Wrangler's build command runs
scripts/build.sh.
@guybedford
Mount the world's SQLite filesystem and write stdio through NODERAWFS
cb0dae9 
The world is again a live SQLite-backed filesystem instead of a tree copied
to and from storage at start and stop: the object constructor mounts
durable-object-fs at /data through worker-fs-mount, and src/workerd.js
rebinds Emscripten's NODERAWFS to that node:fs implementation, so every
Pumpkin write is durable as it happens. Relative paths are resolved against
Emscripten's own working directory, which is the object's rather than the
isolate's. The restore/checkpoint code and the files table go away; status
reports the last save time.

The stdio override is removed: workerd's node:fs supports the process's
stdout and stderr descriptors, and Wrangler relays them as plain stdout and
stderr lines rather than console errors.

Wrangler's build command runs scripts/build.sh so it resolves the
provisioned worker-build regardless of PATH.
@guybedford
Run on Wrangler's bundled workerd
e65b513 
workerd 1.20260918.1 ships the net.Server Durable Object routing and the
node:fs fixes this server needs, so Wrangler 4.135.0's bundled workerd is
used as is; the MINIFLARE_WORKERD_PATH instructions for running a workerd
main build go away. The version is also pinned directly in package.json.
@guybedford
Bring in the memory and configuration work from dlapid/moreMemoryOpts
a244fb5 
Ports Dan Lapid's dlapid/moreMemoryOpts (ec02ab2, cd83251) onto the event-loop
runtime, keeping what applies and dropping what the runtime change superseded.

Pumpkin patches: packed chunk palettes with copy-on-write light arrays,
on-demand carving masks, bounded density pools, pruned structure caches and
lazily grown pathfinder heaps (pumpkin-memory.patch); byte-bounded outgoing
chunk batches with backpressure, idle generation-graph release, and
feature-gated imports so headless builds compile without warnings
(pumpkin-emscripten.patch). The injected-stream connection entry point, the
checkpoint barrier through the chunk writer, and the inline Rayon shim are
not taken: the stock accept loop, the mounted filesystem and Tokio-spawned
jobs cover them.

Worker: operator settings (view and simulation distance, player limit,
compression, MOTD, seed, idle timeout) are Worker variables, validated in
the object's constructor and reported in status. The `#[event(connect)]`
handler reads the Minecraft handshake over the `worker::Socket`: server-list
pings are answered from the object's metadata without starting a server, and
login connections are forwarded with `Stub::connect`, the consumed bytes
written ahead and both directions copied with Tokio. The object's `connect`
hands each socket to Pumpkin's listener with
`Socket::handle_as_node_connection`. After the last player leaves the server
saves the world and stays up for the reconnect window before stopping;
status gains runtime_starts, idle_deadline and settings. Packet compression
is on by default.

worker-fs-mount rejected the S_IFREG | perms mode Emscripten's FS.open
passes to its backend, so a player's data file could not be created on
disconnect; npm install now applies patches/worker-fs-mount-open-mode.patch
(masking numeric modes to their permission bits, as Node does) until
danlapid/worker-fs-mount#10 ships.

Tests: compression-aware protocol client, unit tests for framing, an
experience probe for measuring configurations, and an integration flow
covering pings that must not start the runtime, the player limit, a
reconnect during the idle window and restart without compression. Docs:
configuration guide with the measurement results.
@guybedford
Declare the Durable Object under exports and move to Wrangler 4.141.0
ada9b00 
The deployed Worker was last uploaded with the declarative `exports`
configuration, which the API does not allow reverting to `durable_objects`
bindings and `migrations`, so wrangler.jsonc declares `MinecraftWorld` as a
SQLite durable-object export and the Worker reaches its namespace through
`cloudflare:workers`' `exports` rather than an environment binding.

Wrangler 4.141.0 with workerd 1.20260925.1, and setup runs `npm ci` so one
command provisions everything.
@guybedford
Restore Wrangler's build hook and take worker-build's macOS wasm-opt fix
69e4c87 
wrangler.jsonc had its build hook renamed away by a probe run, so `wrangler
dev` and `deploy` were not rebuilding Rust; the hook is back. worker-build
(cloudflare/workers-rs#1061) now runs wasm-opt single-threaded on macOS
itself, so the build script no longer does.
@guybedford
Bring the docs and object comments up to date with the current runtime
45f16bb
@guybedford
Take over the /data mount in each object constructor
2588725 
worker-fs-mount's mount table is per isolate, and in production an isolate
outlives an object instance: constructing the object again (after eviction,
or another world landing in the same isolate) threw "Path already mounted".
The constructor now unmounts first; one wasm instance serves one object at a
time, so the current object owns the mount.
@guybedford
Move to the worker and worker-build 0.8.7 releases
483506b 
cloudflare/workers-rs#1061 has landed and shipped: the `worker` crate comes
from crates.io and setup installs the matching worker-build release. The
Worker's CPU limit is raised to 3,000,000 ms for world generation.
@danlapid
danlapid merged commit 8c4b282 into danlapid:main Sep 26, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@guybedford @danlapid