Skip to content

Swap tier: broad coverage, copy-back before set_vprot, opt-in sparse JIT dump - #179

Merged
willfaust merged 5 commits into
willfaust:mainfrom
spitefulowl:memory-swap-broad
Oct 4, 2026
Merged

Swap tier: broad coverage, copy-back before set_vprot, opt-in sparse JIT dump#179
willfaust merged 5 commits into
willfaust:mainfrom
spitefulowl:memory-swap-broad

Conversation

@spitefulowl

@spitefulowl spitefulowl commented Oct 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Why

iOS kills the app at a 4 GB footprint. The file-backed swap tier (guest memory mapped from a file, which jetsam does not charge) only backed fresh large commits. Memory that a game commits in small pieces inside a big reservation, or keeps below the guest band, stayed anonymous: Unity and Red Dead Redemption 2 reserve big and commit small, and Metro 2033 Redux's heap never reaches the guest band.

The tier stays off by default (it is enabled by swap-mb >= 64 in madeira.cfg), and classic stays the default coverage.

Rebased onto main: the JIT-pool dump function now sits after #136's ios_x18_derived_base. Nothing else changed.

Commits

  1. Copy swap-tier pages back before set_vprot, not after (ml1257). The copy-back to anonymous memory ran after the new protection was set. As a result, an EXEC request could fail VirtualProtect with STATUS_ACCESS_DENIED before the copy-back ran, and a guard page could come back writable. The copy-back now runs first. This affects every coverage, but only ranges the tier has backed.
  2. Broad swap coverage (ml1257, ml1258, ml1226, ml1221), selected with env.MADEIRA_SWAP_COVERAGE = broad or swap-mode = 2:
    • every new writable reservation of 4 MB to 8 GB below FEX's band (except the JIT pool and the FEX arena) is backed whole when it is made;
    • a decommit inside it punches a hole in place, so the backing survives decommit/recommit;
    • the file becomes a 128 GB sparse offset space, and swap-mb caps its disk use;
    • a churn filter keeps sizes whose backed blocks die within 3 s anonymous;
    • swap-min-mb sets the floor of every coverage;
    • the stats and census lines say what was backed and why not.
  3. Settings › Memory & sync offers broad coverage. It writes classic explicitly when large allocations are chosen while the cfg selects broad.
  4. JIT pool 256 MB and video memory 512/1024 MB choices (ml1241). These are the lower ends that the launch and winemetal already accept.
  5. JIT-pool dump only with MADEIRA_JIT_DUMP, and sparse (ml1242). On the first unhandled Mach exception or SIGILL, ntdll wrote the whole 896 MB JIT-pool alias to Documents. Reading untouched pages charged ~600 MB to the footprint, and Hollow Knight was jetsammed four seconds later. The dump is now opt-in and skips never-written pages.

Testing

  • Metro 2033 Redux (swap-mb 3072, swap-min-mb 4, broad):
    • stable for ~200 s including a level load, with 0 punch failures;
    • ~3.6 GB of reservations file-backed using ~630 MB of disk;
    • footprint peak 2717 MB, against ~3.3 GB without the tier.
  • Ori and the Will of the Wisps with the churn rule: 494 s without jetsam (peak 3981 MB, 1661 MB file-backed). The earlier churn rule took the long-lived Unity heap out of the tier and was jetsammed at 269 s.
  • Host tests:
    • check-swap-coverage.py has a fixed stale tier-start expectation. It is Linux-only as written; on macOS it passes with only its platform shims swapped. It has no case for broad, churn or punch-in-place yet.
    • check-runtime-settings, check-frontend, check-library-sections and check-library-api pass.
    • ConfigCatalog is regenerated and current.
  • Not measured on their own: the copy-back order (the guard-page case was not observed), the new Settings choices and the sparse dump. All three are in device builds.

Known limits

  • With Hold released 1-16 MB guest allocations for 2 s before unmapping #146's MADEIRA_FREE_DELAY_MS (opt-in), a released 1-16 MB guest-band block is unmapped only after the delay, and the churn filter reads the clock at the unmap. With a 2000 ms hold its 3 s rule then acts like a ~1 s rule for those sizes. Nothing changes while either feature is off, and no game has run with both.
  • The disk cap is soft. Dirty file pages that are not yet written back are not in st_blocks, and a reservation backed before the cap was reached keeps committing.
  • The punch and the existing decommit edge memsets run under virtual_mutex. In a backed reservation the edge pages are file pages, so a page-in can happen under the lock.

Notes

No binary is included. libntdll_unix.a is built by build/ntdll-unix/build.sh.

🤖 Generated with Claude Code

spitefulowl and others added 5 commits October 4, 2026 13:50
@spitefulowl @claude
ntdll: copy swap-tier pages back before set_vprot, not after
d3b87db 
set_protection() moved a range out of the file-backed swap tier (the ml1077
copy-back to anonymous memory) only after set_vprot() had applied the new
protection. Two things went wrong with that order:
- an EXEC request was first applied as an mprotect of the shared file
  mapping, so that mprotect decided whether VirtualProtect succeeded; if it
  failed, set_protection() returned STATUS_ACCESS_DENIED and the copy-back
  never ran;
- the copy-back's anonymous RW remap ran after the new protection was set,
  so a guard page could come back writable.

The copy-back now runs first. It re-applies the pages' current (old)
protections to the anonymous copy, and set_vprot() then applies the new ones
to anonymous memory. The ios_swap_release_range() comment says so. This
affects every swap coverage, not only the broad one, and only ranges the
tier has backed (the tier is off unless madeira.cfg swap-mb >= 64).

Tagged ml1257 (with the ml1077 tier code).

Status: ran on the device as part of the broad swap build (Metro 2033
Redux, swap log 1 of 2026-09-27: stable, no copy-back failures); the guard
case itself was not observed on the device. Compiles alone
(clang -fsyntax-only with the build/ntdll-unix/build.sh flags).

Rebuild: libntdll_unix.a (build/ntdll-unix/build.sh); nothing committed.

Renumbered from ml1158 to ml1257: upstream uses those numbers for other
work. Device logs from before the renumbering show the old tags.

Signed-off-by: spitefulowl <spitefulowll@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@spitefulowl @claude
ntdll: broad swap coverage, whole reservations backed when made
b3ef353 
The file-backed swap tier (ml1077: guest memory mapped from a file, which
iOS does not charge to the jetsam footprint) backed only fresh commits: the
classic coverage takes single 8 MB+ commits in the guest band, blocks and
wide take 1 MB+ commits. Memory a game commits in small pieces inside a big
reservation, or keeps below the guest band, stayed anonymous and counted
against the 4 GB jetsam limit. Metro 2033 Redux's heap never reaches the
guest band (`[phys-map] guest=0/0`); Unity and Red Dead Redemption 2
reserve big and commit small.

A fourth coverage, broad (env.MADEIRA_SWAP_COVERAGE = broad, or madeira.cfg
swap-mode = 2 when the env key is unset):
- every new writable VirtualAlloc reservation of at least the floor (4 MB,
  madeira.cfg swap-min-mb) and at most 8 GB, anywhere below FEX's band
  except the JIT pool and the FEX arena, is backed WHOLE when it is made,
  PROT_NONE where not committed, so every later commit inside it, of any
  size, is an mprotect of file pages (ios_swap_whole_resv, called first in
  allocate_virtual_memory, never for ARM64EC code or force_exec_prot);
- a decommit wholly inside a reservation extent punches the file range in
  place (F_PUNCHHOLE; a hole reads as zero) instead of replacing it with
  anonymous memory, so the backing survives decommit/recommit; wide's
  reservations get the same (that path has not run on a device);
- the file becomes a 128 GB sparse offset space and swap-mb caps the disk
  it occupies (fstat st_blocks), checked before each new backing;
- churn filter (ml1258, rule of ml1226): Metro 2033 Redux reserved and
  freed 4 and 16 MB blocks ~9 times a second (1834 backings against ~170
  that lasted, every touched page a file fault, every free a 4-16 MB hole
  punch). A size whose backed blocks die within 3 s at least 8 times, and
  in at least half of its backings, stays anonymous from then on. ml1258's
  first rule (two young deaths) had taken ~1 GB of Ori and the Will of the
  Wisps' long-lived Unity heap out of the tier;
- swap-min-mb sets the floor of every coverage (MADEIRA_SWAP_MIN_KB still
  wins for blocks, wide and broad);
- the [decommit-zero] swap-hole line is rate-limited (4301 lines per run).
The stats line gains disk use, reservations, holes, disk-full refusals and
churn counts, and the census runs with every stats line (~10 s) and lists
the bytes of fresh reservations not taken whole, by reason (ml1221; plain
counters and a stack buffer). Backings of 64 MB or more are always logged.

The tier stays OFF by default (swap-mb >= 64 enables it) and classic stays
the default coverage. The config catalog (gen-config-catalog.py overlay,
regenerated) lists broad as a coverage choice and gains swap-mode and
swap-min-mb; the Memory & sync picker follows in the next commit.

Known limits, from review:
- the disk cap is soft: dirty file pages not yet written back are not in
  st_blocks (scratch test on macOS/APFS: 64 MB dirtied, st_blocks 0 until
  msync), and a reservation backed before the cap was reached keeps
  committing, so disk use can exceed swap-mb; the comments and docs say so;
- the punch and the pre-existing decommit edge memsets run under
  virtual_mutex; in a backed reservation the edge pages are file pages
  (a page-in under the lock, the ml1081 hazard), more often with broad.

Tagged ml1257, ml1258, ml1226, ml1221.

Status: verified on the device. Metro 2033 Redux (swap log 1, 2026-09-27,
swap-mb 3072, swap-min-mb 4): stable for ~200 s with a level load, 0 punch
failures, ~3.6 GB of reservations file-backed with ~630 MB real disk use,
footprint peak 2717 MB (~3.3 GB without swap). Ori and the Will of the Wisps
with the ml1226 rule (run 8, 2026-10-02): 494 s without jetsam, footprint
peak 3981 MB, 1661 MB file-backed; run 7 under the two-deaths rule had 1170
MB file-backed and was jetsammed at 269 s.
tests/host/check-swap-coverage.py: the harness stubs the two helpers
defined after the core, and the tier-start expectation is fixed (stale since
upstream 046af3e made classic the default; it failed on main too). The test
is Linux-only as written; run on macOS with only its platform shims swapped
(native F_PUNCHHOLE, mach_vm_region) it passes, and on main it fails only
that stale line. It has no case for broad, churn or punch-in-place yet.
clang -fsyntax-only with the build.sh flags: no new warnings.

Rebuild: libntdll_unix.a (build/ntdll-unix/build.sh); nothing committed.

Renumbered from ml1158, ml1160 to ml1257, ml1258: upstream uses those
numbers for other work. Device logs from before the renumbering show the
old tags.

Signed-off-by: spitefulowl <spitefulowll@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@spitefulowl @claude
Library: offer broad swap coverage in Settings > Memory & sync
06c5f15 
The Swap coverage picker offered classic, blocks and wide only, so the
broad coverage of the previous commit could be chosen only by editing
madeira.cfg. It now offers "Whole reservations 4 MB+ (broad)".

madeira.cfg swap-mode = 2 also selects broad when env.MADEIRA_SWAP_COVERAGE
is unset, so the picker shows broad in that case (swapModeBroad), and
choosing large allocations then writes `classic` explicitly instead of
removing the key (which would have meant broad again). The help text and
docs/LIBRARY.md describe broad and that swap-mb limits its disk use (a soft
cap, checked when a block is backed). Catalog regenerated (swap-mode is now
also read by Library.swift).

Tagged ml1257.

Status: the broad coverage itself is verified on the device (see the
previous commit); the picker shipped in the device builds since the
2026-09-29 rebase, without a recorded issue. Host tests check-runtime-settings,
check-frontend, check-library-sections, check-library-api and
check-config-catalog (generator part run against the fork submodules) pass.

Renumbered from ml1158 to ml1257: upstream uses those numbers for other
work. Device logs from before the renumbering show the old tags.

Signed-off-by: spitefulowl <spitefulowll@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@spitefulowl @claude
Library: JIT pool 256 MB and video memory 512/1024 MB choices
7627567 
Settings > Memory & sync offered a JIT pool of 512 MB at the least and video
memory of 1536 MB at the least, although the launch accepts a madeira.cfg
pool of 256 to 1152 MB and winemetal any vram-mb of 256 or more. The pickers
now also offer a 256 MB pool and 512 and 1024 MB of video memory, the
lower ends of what the launch and winemetal accept (owner's request).

Tagged ml1241.

Status: built into a device build (2026-10-03); no device report on the
new choices yet. Host tests check-runtime-settings and check-config-catalog
pass; the catalog does not change (the choices are hand-built rows).

Signed-off-by: spitefulowl <spitefulowll@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@spitefulowl @claude
ntdll: JIT-pool dump only with MADEIRA_JIT_DUMP, and sparse
bc46c48 
On the first unhandled Mach exception and on the first SIGILL, ntdll wrote
the whole JIT-pool RW alias (896 MB) to Documents/fex-jit-dump.bin. The
production pool is not no-footprint, so reading an untouched page of it
materialises a real zero page: the dump alone charged ~600 MB to the
footprint. Hollow Knight went from 2169 to 2755 MB while it ran and was
jetsammed (4096 MB limit) four seconds later.

The two copies of the dump code become one function, ios_dump_jit_pool().
It does nothing unless MADEIRA_JIT_DUMP is set, and it skips the pages that
mincore() reports neither resident nor paged out (never written): they stay
holes in the file, so file offsets still equal pool offsets. The getenv line
carries a comment, which becomes the switch's catalog note (regenerated).

Tagged ml1242.

Status: built and in the device builds since 2026-09-25; Hollow Knight
reached gameplay afterwards, but no log compares the footprint before and
after. The cost it removes was measured on the device (jetsam triage of
2026-09-25). clang -fsyntax-only with the build.sh flags: no new warnings.

Rebuild: libntdll_unix.a (build/ntdll-unix/build.sh); nothing committed.

Renumbered from ml1140 to ml1242: upstream uses those numbers for other
work. Device logs from before the renumbering show the old tags.

Signed-off-by: spitefulowl <spitefulowll@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@willfaust
willfaust merged commit 5fa0b2f into willfaust:main Oct 4, 2026
willfaust added a commit that referenced this pull request Oct 4, 2026
@willfaust @claude
Fixes on spitefulowl's app, swap and audio pull requests
791aa6b 
- metal-validation (#182) is read through madeira_cfg.h, so the settings
  catalog lists it.
- A game's fastsync semaphore choice (#181) is exported only when the game
  made one, so madeira.cfg's env.MADEIRA_FASTSYNC_SEM still applies;
  MADEIRA_CPU_COUNT and DXMT_D9_ANISO_LIMIT are unset when a game does not
  choose them, so a previous game's value never beats madeira.cfg.
- The JIT-pool dump (#179) stays on by default, sparse; MADEIRA_JIT_DUMP=0
  turns it off. The 256 MB pool choice says large games can run short.
- MADEIRA_AUDIO_LEAD_MS=0 (#180) restores the plain 10 ms beat without the
  time-constraint policy; a stream that leaves two wake-ups unanswered is
  polled on the 10 ms beat instead of every 1 ms.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@spitefulowl @willfaust