Skip to content

LLVM: exit to dispatcher when mtmsr enables MSR[EE] - #23

Closed
dougchansan wants to merge 4 commits into
ExpansionPak:mainfrom
dougchansan:fix/ee-delivery
Closed

LLVM: exit to dispatcher when mtmsr enables MSR[EE]#23
dougchansan wants to merge 4 commits into
ExpansionPak:mainfrom
dougchansan:fix/ee-delivery

Conversation

@dougchansan

Copy link
Copy Markdown
Contributor

Summary

Pokemon Colosseum (GC6E01) rendered at 0.25 fps under the LLVM backend against 30 fps under the C backend, and could not cold-boot past its THP intro videos. Root cause: external-interrupt delivery starvation. This PR makes an mtmsr that enables MSR[EE] exit to the dispatcher, creating the delivery point the block-ending JITs get for free. The runtime half is ExpansionPak/RecompCore PR (delivers a pending external interrupt when a dispatch boundary shows EE=1); each half is safe alone, both are needed for the fix.

The mechanism

  • Colosseum waits for VI retrace by yield-spinning: OSDisableInterrupts -> SelectThread(yield) -> OSRestoreInterrupts, check callback flag, check a 3-second bus-clock timeout, repeat. The other tested titles block on wait queues; this spin is why only Colosseum failed.
  • Under the LLVM backend the whole critical section runs inside native calls within a single dispatch. The only burst-ending points are the budget guards, which sit at call sites -- in OS code, almost entirely inside interrupt-disabled windows. Burst boundaries phase-lock to EE=0.
  • External interrupts are only deliverable when the host regains control with EE=1. Measured on the pinned scene: the VI interrupt was pending-but-blocked at 30% of delivery checks (C: 0.76%), and only 1.2% of raised interrupts were ever delivered. The post-retrace callback never ran; every frame escaped through the game's own 3-second timeout -- 1 frame / ~4 s = 0.2487 fps exactly.
  • mtmsr was lowered as a plain state write. Dolphin's Jit64/JitArm64 end the block at mtmsr and check external exceptions; this change mirrors that: after the MSR store, a 0->1 transition of EE takes a side exit (sync, store pc+4, return). mtmsr is supervisor-only, so the exit is rare and cheap.

Measurements (Windows, pinned savestate, same runtime build)

before after
Colosseum LLVM 0.2487 fps 27.4-28.8 fps (4 runs)
Colosseum C (control) 29.98 29.99
MKDD LLVM (control) 48.65 51.0
Colosseum LLVM cold boot hangs at THP videos 54.8-55.5 fps, boots

Interrupt counters on the pinned scene: raised 484 -> 31,777; delivered 462 -> 29,856; pending-but-blocked checks 30% -> 0.5% -- matching the healthy C profile.

Investigation history: #20.

External interrupts are only deliverable when the host regains control at
a dispatch boundary with MSR[EE] set. Generated code runs whole critical
sections between boundaries, and the budget guards that end a burst sit
at call sites, which in OS code fall almost entirely inside
interrupt-disabled windows. A guest that waits by yield-spinning
(enable, check, disable, reschedule) then never shows an enabled window
at a boundary and pending interrupts starve.

Pokemon Colosseum waits for VI retrace exactly that way and rendered one
frame per 3-second timeout (0.25 fps) while the C backend ran at 30 fps.
With this exit (and the matching runtime change that delivers a pending
external interrupt at an EE=1 boundary) it runs at 28 fps and cold-boots
past its THP intros. Mario Kart Double Dash is unaffected (48.7 -> 51.0).

mtmsr is a supervisor instruction on OS paths only; the exit fires just
on a 0->1 EE transition, mirroring the block-ending JITs.
@dougchansan

Copy link
Copy Markdown
Contributor Author

Backend matrix on Colosseum, now that it runs

All five arms on the pinned Phenac City scene (part1-phenac-city.sav), Windows, same runtime build, two runs each. The mtmsr delivery exit was ported locally to the old and AOT emitters for this comparison (their DOLIR_OP_STATE_WRITE handling differs; local branches only) -- without it they still read 0.25 fps.

arm run 1 run 2 vps
C backend 29.99 29.99 59.9
LLVM old + fix 25.4 27.3 50-54
LLVM new (this base) + fix 25.1-29.0 across runs 50-58
LLVM new + --state-in-memory + fix 29.97 29.97 59.9
LLVM AOT + fix 29.97 29.97 59.9

Reading the numbers honestly: Colosseum is retrace-locked at 30 fps -- the game's frame loop waits for the VI callback -- so fps here measures saturation, not backend speed, and no arm can beat C's 29.99. The signal is which arms hold the cap:

  • state-in-memory and AOT hold 59.9 vps / 29.97 fps in every run.
  • Plain new and old LLVM run 8-16% under cap. The overhead is measured, not guessed: each EE-enabling mtmsr side exit resumes at pc+4, which is mid-block, and the dispatcher covers the rest of the block (two instructions of OSRestoreInterrupts) through per-instruction interpreter fallback -- ~120M hook-fallback instructions per run, 59M of them the rlwinm/blr pair at 0x8009DF80/84. With state-in-memory the sync cost of those crossings collapses, which is why that arm saturates.

Two attempted optimizations are documented as not viable as-is: saturating the burst budget instead of exiting (deliver at the next guard), and marking pc+4 a region leader so the resume is native. Both build and both wedge Colosseum deterministically ~10-20 s into the scene, in an audio-handler interrupt loop -- delivery placement interacts with the AI/DSP handler's enable-interrupts-around-callback pattern in a way that needs its own investigation before optimizing this path. The shipped fix is the variant that has survived every run, the control titles, and a cold boot.

@dougchansan

Copy link
Copy Markdown
Contributor Author

Incorporated upstream: emitStateWrite in 1bec355 implements this exit (msr_ee_exit at pc+4 on an EE 0->1 transition). Closing as superseded. Two follow-ups remain live: ExpansionPak/RecompCore#18 is the required runtime half -- without it the boundary exists but pending external interrupts still wait for slice granularity and Colosseum starves back to 0.25 fps -- and #24 tracks a large throughput regression measured on current main relative to this branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant