LLVM backend: Colosseum does not get through boot (x86-64 and aarch64; C backend does)

[dougchansan]

Correction (updated): I originally filed this as AArch64-specific. That was wrong. The failure reproduces on x86-64 with the same DOL and the same harness, so this is a Colosseum + LLVM-backend defect, not an ARM one. Details and measurements below.

Building Pokémon Colosseum (GC6E01) with the LLVM backend produces a module that does not get through the boot sequence. The C backend, same DOL, same runtime, same automation harness, boots normally.

Measurements

Cold boot, A-press driver, no savestate, identical harness on each platform:

platform	backend	frames produced	fps	speed
x86-64 Windows	C	4262	44.77	0.749
x86-64 Windows	LLVM	16	0.26	1.00
aarch64 (Pi 4)	C	1417	9.74	0.166
aarch64 (Pi 4)	LLVM	8	5.74 (stale)	0.634

The guest is not deadlocked — it executes (speed is nonzero, the VI keeps ticking), it just never makes forward progress. Note the LLVM arm reports a higher speed than C on the Pi while producing no frames: it is spinning cheaply instead of doing work.

Luigi's Mansion (GLME01) with the LLVM backend runs fine on the same Pi (769 frames, 6.11 fps), so the AArch64 LLVM backend is not broken in general.

Where it ends up

perf on the wedged aarch64 run puts ~58% of samples in three guest functions, which the DOL disassembly identifies as the thread scheduler:

• 0x8009BBE0 — OSSaveContext (saves GQR1–5 via mfspr 0x393..0x397, CR, LR, MSR, CTR, XER)
• 0x8009BC50 — OSLoadContext, which resumes a thread with rfi
• 0x800A17E0 — run-queue enqueue / reschedule

None of these appear in the C backend's profile, which is dominated by real game work (func_800FD5E0, paired-single ops, FP helpers).

What is NOT the cause

I instrumented the shared runtime to rule these out rather than guess:

• Thread resume is correct. ppc_rfi fires ~150M times in 70s (vs ~150k for C, ~1000x more), and at every sampled resume CPUState is correct: srr0=0x800A183C, r3=0x00000001. 0x800A183C is the instruction after bl OSSaveContext, and r3=1 is exactly the "resumed" return value the scheduler tests. The C backend resumes at the same address with the same value, so this churn is a normal idle loop — the LLVM arm is just doing ~1000x more of it.
• Entry at that PC is valid. In the generated IR the address is a real entry case (guest_800A183C_b23), reached from native_entry, and the block loads r3 from CPUState and branches on icmp eq i32 %state2.7, 0 correctly.
• Not an unsupported instruction. All 1643 fallbacks in the generated CSV are embedded-data; there are no unmodeled opcodes.
• Not exception delivery. Exception counts stayed under threshold in both arms.
• Not our open PRs. Regenerating with Register only the LLVM targets this build can emit for, and build the bench on Windows #17/Route fallback blocks through per-edge trampolines so the phi matches the CFG #18/Key the object cache on the sources that generate the objects #19 applied produces byte-identical objects (4890/4890 identical, 0 changed) and an identical module hash, so those fixes neither cause nor fix this.

Still open

I have not isolated the primitive that diverges. The dispatch-address stream diverges from the C backend within the first handful of dispatches at boot, and the LLVM arm reaches far less distinct guest code over a run (909 vs 4000+ distinct addresses in the same window).

Reproduction is easy on x86-64 now, so this no longer needs ARM hardware to chase. Happy to keep digging or hand over what I have.

[dougchansan]

Further evidence, and one hypothesis eliminated.

The DOL was not the cause. Our Windows workspace had been recompiling a modified Colosseum DOL (md5 b299467dd305) while the Pi used the retail one. I re-extracted from the retail disc image with DolphinTool (md5 c55df119fad4, matching the Pi) and rebuilt every arm from that. The defect is unchanged.

Cold boot, x86-64, retail disc data, identical harness per arm:

arm	fps	vps	speed	frame_count	present_count
C	59.94	59.94	1.000	5353	10755
LLVM (upstream rewrite)	0.246	59.94	0.999	8	1575
LLVM (AOT region backend)	0.250	59.94	0.999	8	728

Two details that sharpen the picture:

• The guest runs at full speed (speed ≈ 1.0) with the video interrupt ticking normally (vps 59.94). This is not a slow module — it is a module that does not render.
• present_count keeps rising while frame_count is frozen at 8. The runtime is still swapping buffers; the game is simply re-presenting the same image because its main thread never advances.

It also ruins a mid-game savestate scene, not just boot. Benchmarking part1-in-town.sav with the interleaved harness, all three LLVM backends are degenerate while C is healthy:

arm	samples	fps median	speed median
C control	114	54.67	0.9159
LLVM old (pre-rewrite)	11	0.34	1.3250
LLVM new (rewrite)	17	0.58	2.1241
LLVM aot	26	0.85	3.1201

Worth flagging for anyone benchmarking this title: because speed counts retired guest cycles and this defect burns cycles in a spin that does no work, the metric rewards the broken arm. On speed alone the AOT arm reads as a 3.58x win over C. It is an artifact — fps and sample count expose it. I withdraw an earlier internal Colosseum figure that was measured this way.

So the defect: affects all three LLVM backends (pre-rewrite, rewrite, and AOT), reproduces on x86-64 and aarch64, on retail data, at boot and from a savestate. The C backend is unaffected throughout.

[dougchansan]

Further investigation. No root cause yet, but three useful narrowings and one correction to something suggested along the way.

Correction first

A hypothesis was raised that the C backend avoids the bug because its text0 chunk fails SMC verification and the whole 0x80003100-0x80005500 range therefore runs on Dolphin's JIT rather than recompiled code. That is not what happens. Runtime counters from a C-backend run:

[staticrecomp] shutdown: native=1494658421 fallback=0 native_exc=78253
               hook_fb=127384220 smc_failed=0 verifications=98 reverify_events=5

1.49 billion natively executed instructions, zero fallbacks, zero SMC failures. The C backend is genuinely running recompiled code for this title. Both backends report fallback=0 smc_failed=0, so the difference is not one of them quietly running on the JIT.

New: the divergence is visible in the runtime counters

Same title, same scene, same runtime, C versus LLVM:

counter	C	LLVM
native instructions	549,841,950	181,820,064
native_exc	38,659	521
hook_fb	86,909,490	108,309
fallback	0	0
smc_failed	0	0
guest cycles	13,686,478,575	14,779,848,761

The LLVM arm retires a comparable number of guest cycles while taking 74x fewer exceptions and 800x fewer hook fallbacks. It is not doing the same work. That is a much sharper signal than the frame counter and it is cheap to reproduce.

New: the rendering ratio is 238:1

Measured directly: 238 VI interrupts fire per rendered frame under LLVM, against 1 per frame under C. So the game's render-trigger condition is being satisfied roughly once per 238 VI callbacks rather than every one. That is consistent with the counters above — something the game does every frame is not happening.

Ruled out this round, with evidence

The VI exception stub at 0x800039AC was inspected in the emitted IR and is correct:

• SPRG1/2/3 saves emit ppc_mtspr(ctx, 273/274/275, ...) with the right SPR numbers
• SRR0 set to 0x800C0EAC, SRR1 to old_msr | 0x30
• gpr[2] = interrupted PC, gpr[3] = 0x500, gpr[4] = srr1
• ppc_rfi tail call with the correct CIA
• The alignment stub at 0x80003AAC has the same structure with exception type 0x600

Two earlier hypotheses were also refuted on inspection: 0x7C5143A6 at 0x800039AC decodes as mtspr SPRG1, r2 (correct), and the mfmsr/ori pair lowers correctly to or i32 %117, 48.

Not yet examined

The FPU-unavailable stubs at 0x800044AC, 0x800045AC, 0x800046AC are structurally unlike the others — they save and restore the Condition Register around the handler (mfcr / andi. / sync / mtcrf 0xFF / xoris). A CR save/restore error there would corrupt condition codes across an exception and change branch outcomes in game code, and the FPU-unavailable exception fires during startup before the FPU is enabled. That is the next place worth looking, though it is a suspicion rather than a finding.

Happy to keep digging or hand this over. The counter comparison above is the cheapest reproduction of the divergence I have found so far.

[dougchansan]

More probing. Not resolved, but the divergence now has a number and a direction.

The sharpest signal: the LLVM arm barely takes any exceptions

Runtime counters, same title, same scene, same runtime, comparable guest cycles:

counter	C	LLVM
guest cycles	13,686,478,575	14,779,848,761
native_exc	38,659	521
hook_fb	86,909,490	108,309
bursts	2,712,107	1,132,575
fallback / smc_failed	0 / 0	0 / 0

The LLVM module retires slightly more guest cycles while taking 74x fewer exceptions. External interrupts are how VI reaches the guest, so a title that renders nothing while running at full speed with the host VI ticking at 59.94 is consistent with interrupts not being delivered rather than with slow code.

That also matches the independently measured 238 VI interrupts per rendered frame.

What the working arm actually spends its time on

The runtime has a built-in dispatch sampler (STATICRECOMP_DISPATCH_SAMPLES=1) that records dispatch PCs and dumps the top sites at shutdown. On the C backend the top two are:

dispatch-site pc=8009df3c samples=51999
dispatch-site pc=8009df64 samples=51576

0x8009DF3C disassembles to:

8009DF3C  mfmsr  r3
8009DF40  rlwinm r4, r3, 0, 17, 15     ; clear MSR[EE]
8009DF44  mtmsr  r4
8009DF48  rlwinm r3, r3, 17, 31, 31    ; return previous EE
8009DF4C  blr

That is OSDisableInterrupts(), with OSRestoreInterrupts() immediately after it at 0x8009DF50. The busiest thing the working backend does is enter and leave critical sections — which is exactly the machinery that governs whether interrupts get delivered.

Note the LLVM arm could not be sampled the same way: it never processes the automation stop command (waited 40s), so its counters never flush. That inability to shut down cleanly is itself consistent with the emulation loop being stuck.

Ruled out this round

mtmsr lowering is not the difference. Both backends do the same thing:

• C: ctx->msr = ctx->gpr[rS]
• LLVM: write_slot(DOLIR_STATE_MSR, gpr(rS)) via the IR builder

Keeping guest state in memory does not fix it. I built the title with the --state-in-memory change from #21, which removes the entry prologue, the materialization barriers and the post-call reloads, so MSR is written to CPUState immediately rather than held in a promoted SSA value. If delayed visibility of MSR[EE] were the cause, that should have changed the outcome. It does not:

arm	fps	frame_count
C	59.94	5353
LLVM (current)	0.246	8
LLVM + --state-in-memory	0.249	8

Identical failure. So the defect is not related to state hoisting.

Gather pipe is not obviously implicated. The static recomp core routes 0xCC008000 page writes straight to GPFifo, and GPFifoManager::Write32 self-kicks via CheckGatherPipe().

Where I would look next

The native_exc gap is the cheapest reproduction of the divergence and does not need a benchmark run — boot either module for 70s and read the shutdown line. Anything that explains why the LLVM module takes 521 exceptions where C takes 38,659 over the same workload very likely explains the rendering failure too.

The suggestion that this title is stuck on its THP intro video is consistent with everything above, though I have not confirmed it directly: a video that never advances would look exactly like this if the interrupts driving it are not arriving.

[dougchansan]

Ran the runtime's lockstep verifier against this title, and then ran the same thing against a title that works. Posting both because the second invalidates the first as evidence.

Lockstep on Colosseum looked like a smoking gun

Built the module with DOLRECOMP_LLVM_WRITE_JOURNAL=1 and ran with STATICRECOMP_LOCKSTEP=1. It diverges from Dolphin's core at the first checked region:

DIVERGE #1 entry=0x80003158 end=0x80005498:
  r3:  N=0x6f05      I=0x6f09
  r4:  N=0x8039a77c  I=0x8039a6fc
  xer: N=0x20000000  I=0x0
  mem[0x8048e722]: N=0x20 I=0x00

0x20000000 is XER[CA]. r3 is 4 lower and r4 is 0x80 higher, which is exactly 4 iterations of the 0x20-stride block-fill loop at 0x80005498. Reproducible, and unchanged when the step cap is raised from 512 to 65536, so it is not a comparison-window artefact.

But a working title diverges identically

Mario Kart Double Dash renders fine under the same LLVM backend (50.0 fps, 5271 frames in the same 90s window). Its lockstep output:

DIVERGE #1 entry=0x80003158 end=0x80003500:
  mem[0x803e22e2]: N=0x20 I=0x00
  mem[0x803e22e3]: N=0x32 I=0x00
DIVERGE #2 entry=0x80003500 end=0x80003500:
  r3:N=0x3288,I=0x32a1  r4:N=0x803642dc,I=0x80363fbc

Same entry PC, same byte values (0x20, 0x32) at the analogous address, same block-fill loop signature with r3/r4 offset by a whole number of iterations. 25 divergences reported for each title.

So lockstep divergences on this codebase are not by themselves evidence of a miscompile. They appear on a title that works perfectly. The comparison runs native code a region at a time against a stepwise interpreter, and the two are simply not sampled at equivalent points inside tight loops. Anyone using this tool here should calibrate against a known-good title first — I nearly reported the XER carry difference above as the root cause.

What still distinguishes the two titles

The counter gap remains the only signal I have that separates a working title from this one:

counter	C	LLVM
guest cycles	13,686,478,575	14,779,848,761
native_exc	38,659	521
hook_fb	86,909,490	108,309

74x fewer exceptions over slightly more guest cycles, with the host VI ticking normally at 59.94. That is still the cheapest reproduction and still the most likely place the answer lives.

Also confirmed this round: the module is genuinely loaded and executing recompiled code, not silently falling back. native=181820064 fallback=0 smc_failed=0, module hash matches the retail DOL.

[dougchansan]

Broke the exception counter down by vector. This is the sharpest split so far.

Exceptions by vector, same title, same scene, 70s

vector	C backend	LLVM backend
0x800 FP unavailable	24,488	1
0xC00 system call	59,512	19
total	84,000	20

Not a scaling difference — the LLVM module takes twenty exceptions where the C module takes eighty-four thousand, over comparable guest cycles at full speed.

The first 87 exceptions are identical

Logging the vector and SRR0 of every exception and diffing the two sequences: they match exactly for 87 exceptions, then diverge. Both arms then continue hitting the same two sc sites, but the LLVM arm's rate collapses (314 logged versus the 5000 cap for C).

The two system-call sites are both cache/sync:

8009B2E8  mtctr r4
8009B2EC  dcbf  0, r3          ; DCFlushRange loop
8009B2F0  addi  r3, r3, 0x20
8009B2F4  bdnz  0x8009b2ec
8009B2F8  sc                   ; SRR0 = 8009B2FC
8009B2FC  blr

80098034  sc                   ; SRR0 = 80098038, bare sync call
80098038  blr

Tested and ruled out this round

Cache-op handling is not it. dcbf is one of the few places the backends genuinely differ: the C backend emits ppc_fallback_instruction and leaves native execution for every one, while the LLVM backend handles it inline through ppc_cache_control. That accounts for hook_fb being 86.9M on C versus 108K on LLVM, and it was a good candidate for starving the runtime of re-entry points.

Forcing exactly those two functions through the JIT with STATICRECOMP_FALLBACK_RANGES=8009B2E0-8009B310,80098030-80098040 changes nothing: 0.2498 fps versus 0.2467 unforced, frame_count 8 either way.

HookCacheControl is installed and correct — ICBI invalidates the iCache and the JIT, so cache operations are not silently dropped on the LLVM path.

The number I cannot explain

FP unavailable: 24,488 versus 1.

The C backend emits a check before every FPU-using instruction (if (ppc_op_uses_fpu(inst->op)) → ppc_fp_available_inline). The LLVM backend checks once per region and caches the answer in fp_available_checked_, which is reset at region entry and by reloadUsedState().

I found and fixed two genuine cache-invalidation holes while chasing this (an mtmsr write not invalidating the cache, and the native call-resume path reloading values without dropping the emitter's conclusions — both in #22). Neither moves this title: together they change one of 4890 emitted objects.

So the caching holes are real but are not what suppresses 24,487 FP-unavailable exceptions. Something else is preventing the LLVM module from ever taking that exception, and the guest's lazy FPU context switching is built on receiving it. That seems the most promising thread for anyone with more context on this backend than I have.

Everything eliminated so far

SMC/JIT fallback (both run fully native, fallback=0 smc_failed=0), entry dispatch, rfi/thread resume (CPUState correct at every resume), mtmsr lowering, state hoisting (--state-in-memory is byte-for-byte the same failure), gather pipe, cache ops, DOL variant (retail disc data reproduces it), and lockstep divergences as evidence (a working title shows the same ones).

[dougchansan]

Correction to my last comment, and a much better number.

I suggested the FP-unavailable disparity (24,488 versus 1) was the most promising thread. It is a symptom, not a cause. Probing MSR at every rfi shows why.

MSR[FP] at thread resume

	C	LLVM
rfi resumes in 60s	~800,000	91,799,710
MSR[FP] set	742,467	91,799,710
MSR[FP] clear	57,533 (7.7%)	290 (0.0003%)
ppc_fp_available helper calls	0	1

The C arm resumes threads with FP disabled 7.7% of the time — ordinary lazy-FPU behaviour, and each of those is an FP-unavailable exception waiting to happen. The LLVM arm resumes with FP disabled 290 times out of 91.8 million.

So the LLVM module never takes FP-unavailable because it never runs a thread that lacks the FPU. That follows directly from the real problem: it is performing 115x more thread resumes than the C backend while doing almost no work.

What that rules out

The FP-availability caching in the LLVM emitter is not what suppresses those exceptions. I found and fixed two genuine invalidation holes there (#22) and they change one of 4890 emitted objects on this title — consistent with this measurement, not with them being the cause.

Also ruled out this round: forcing the DCFlushRange/sync functions through the JIT with STATICRECOMP_FALLBACK_RANGES=8009B2E0-8009B310,80098030-80098040 changes nothing (0.2498 fps versus 0.2467 unforced). So the inline-versus-fallback handling of dcbf is not it either, despite being one of the few genuine behavioural differences between the backends (hook_fb 86.9M on C versus 108K on LLVM).

Where this leaves it

The guest is stuck resuming a thread ~92 million times in 60 seconds, at full speed, taking essentially no exceptions and making 19 system calls where the C backend makes 59,512. Every resume carries the correct state — SRR0 0x800A183C, r3=1, which is the "resumed" return value the scheduler tests — and the emitted IR for that entry point is correct.

So a thread is being resumed correctly and immediately blocking again, forever, without asking the OS for anything. What it is waiting on is still unknown, and that is the question I would put to anyone with more context on this runtime.

Cheapest reproduction remains the shutdown counter line after a 70s boot: native_exc 521 versus 38,659, no benchmark needed.

[dougchansan]

Colosseum (GC6E01): what the blocked thread is waiting on

Pinned state part1-phenac-city.sav, Windows, same runtime build for both arms.
Baseline: C = 29.98 fps, LLVM = 0.2487 fps, VI ~59.94 in both.

1. The thread

Guest thread 0x803FBCB0, constant stack chain:

803FBCB0 sp=8048E678 <- 804001F0 <- 800A19B0 <- 800D31E4 <- 801E115C <- 80005DE0 <- 800FE808

0x800D31E4 is the resume point of a 3-second timed wait:

800D31C0  stb   r0, -0x5d8f(r13)   ; clear flag
800D31E0  bl    0x800a1990         ; block / yield
800D31E4  lbz   r0, -0x5d8f(r13)   ; resumes here, re-reads flag
800D31F4  lwz   r0, 0xf8(r31)      ; 0x800000F8 = bus clock
800D31FC  srwi  r0, r0, 2          ; ticks/sec
800D3200  divwu r0, r3, r0         ; elapsed seconds
800D3204  cmplwi r0, 3
800D320C  stb   r30, -0x5d8f(r13)  ; timeout -> set flag itself
800D3218  beq   0x800d31e0         ; else keep waiting

2. What sets the flag

Only one other writer, 0x800D3F50 -- a 3-instruction leaf:

800D3F50  li  r0, 1
800D3F54  stb r0, -0x5d8f(r13)
800D3F58  blr

Registered at 0x800D3CA4 via 0x801C01C8, which stores the pointer into the
VI globals at 0x80456DA8 bracketed by OSDisableInterrupts/OSRestoreInterrupts
-- i.e. VISetPostRetraceCallback.

So the thread waits for a VI retrace interrupt, and escapes only via its own
3-second timeout. 0.2487 fps == one frame per ~4.0 s == the timeout path. The
game is not slow; it is timing out once per frame.

3. Why the retrace never arrives

Added counters to the raise (ProcessorInterface::UpdateException) and delivery
(PowerPCManager::CheckExternalExceptions) paths:

arm	raised	delivered	blocked (EE=0)
Colosseum C	33,037	31,209	37,158
Colosseum LLVM	484	462	1,056,463
MKDD LLVM (control)	28,263	25,198	3,494

raised counts only 0->1 transitions, so the low count is a consequence: the
interrupt sits pending because MSR.EE is stuck at 0. 30% of checks are
EE-blocked under LLVM vs 0.76% under C. MKDD under LLVM is healthy, so this is
not a general backend fault.

4. Where it sits with interrupts disabled

Sampling guest PC on every 64th EE-blocked check:

PC	C	LLVM
800a128c (one-instruction blr -- null thread-switch hook)	10	11,077
800a183c (after bl 0x8009bbd0, in __OSReschedule)	32	5,466

Matching dispatch-site sampling: the LLVM arm spends ~97% of dispatches at
800d31e4 + 800a128c/1930/183c. The C arm's top sites are OSDisableInterrupts
/OSRestoreInterrupts and ordinary game code.

The guest is parked in the OS scheduler's idle path with interrupts disabled.

Ruled out

• Timebase hoisting. Made mftb/DOLIR_STATE_TIMEBASE reads volatile and
  non-cacheable (branch fix/timebase-volatile, module hash changed
  625021ef -> e0e7c371). No effect: 0.2487 fps. The 3-second timeout does
  fire, so the guest clock advances correctly.
• rfi lowering. STATICRECOMP_FALLBACK_RANGES=8009bc50-8009bcc0 forces
  OSLoadContext (ends in rfi) to the interpreter. No change (0.2479 fps).
  syncDirtyState() does flush SRR1 before ppc_rfi, and returnFromBody()
  does not re-sync, so the restored MSR is not being clobbered.
• Earlier rounds: SMC/JIT fallback (fallback=0 in both arms), entry dispatch,
  mtmsr lowering, state hoisting, gather pipe, cache ops, DOL variant,
  FP-available cache holes, lockstep divergence signatures.

Note

STATICRECOMP_FALLBACK_RANGES over 8009df3c-8009df90
(OSDisableInterrupts/OSRestoreInterrupts) or over the whole OS region
80098000-800a4000 hangs outright (fps=0, vps=0, delivered=0). Mixed
native/interpreter execution across MSR manipulation is itself broken, so this
is not a usable bisect tool in that region -- worth a separate look.

Instrumentation to revert

• Source/Core/Core/HW/ProcessorInterface.cpp (g_dr_ext_raised)
• Source/Core/Core/PowerPC/PowerPC_Exceptions.cpp (counters + dr_note_ee0)
• Source/Core/Core/PowerPC/StaticRecomp/StaticRecompCore.cpp (shutdown dump)

[dougchansan]

Lockstep: an msr divergence unique to Colosseum

STATICRECOMP_LOCKSTEP=1, both titles LLVM, both capped at 74 reports:

title	divergences	of which involve msr
MKDD (working control)	74	0
Colosseum	74	1

The one Colosseum msr divergence lands in OSDisableInterrupts:

entry=0x8009E724 end=0x8009DF3C
msr:  N=0x9932  I=0x1032
srr1: N=0x9932  I=0x1030
srr0: N=0x8009df3c  I=0x8009e758
r12:  N=0x800a128c        (the null thread-switch hook)

Decoding: I = ME|IR|DR|RI, EE=0. N = same plus EE(0x8000), FE0(0x800),
FE1(0x100). The native side has taken an exception (srr0/srr1 populated) that
the interpreter has not, and the whole GPR file has diverged with it.

FE0|FE1 being set on the native side ties this to the MSR/FP-exception-mode
path. Not conclusive on its own -- one report against a 74-report cap -- but it
is the only msr divergence across both titles and it is in exactly the function
that gates interrupt delivery.

[dougchansan]

Lockstep, uncapped and filtered to msr

The report cap was never the limit -- STATICRECOMP_LOCKSTEP_MAXREPORT already
defaults to unlimited. The real bound is m_ls_checked, which dedupes by entry
PC, so "74 divergences" meant 74 distinct diverging entry PCs. Added two
options to get past it:

• STATICRECOMP_LOCKSTEP_FILTER=<substr> -- only report diffs containing the
  substring (suppressed reports counted, not printed)
• STATICRECOMP_LOCKSTEP_NODEDUP=1 -- re-check entry PCs instead of once each

Also raised STATICRECOMP_LOCKSTEP_STEPCAP to 100000; most of the original 74
were CTRLFLOW:...steps=512 artifacts of the interpreter running out of steps.

Result: 44 msr divergences, 42 of them one bit

180 s run, Colosseum LLVM, part1-phenac-city.sav:

count	native	interpreter	delta
42	0x9932	0x1932	0x8000 = EE
1	0x9932	0x1032	EE + FP
1	0x3932	0x1030	--

Entry PCs:

count	entry	end
21	0x800FEA98	0x800FEA98
21	0x800FEA98	0x800FEA90
1	0x800B8E28	0x800A128C
1	0x8009E724	0x8009DF3C

42 of 44 come from one loop. 0x800FEA90-0x800FEAE4 walks a linked list at
r13-0x5B88 and bctrls each node's handler at offset 20, then calls
OSDisableInterrupts -- a callback dispatcher running in interrupt context,
where EE should be 0. The interpreter agrees; native has it set.

Why that is plausible as a bug

The OS interrupt primitives are all mfmsr/mtmsr pairs:

8009DF3C  mfmsr r3; rlwinm r4,r3,0,17,15; mtmsr r4    ; OSDisableInterrupts
8009DF50  mfmsr r3; ori r4,r3,0x8000;     mtmsr r4    ; OSEnableInterrupts
8009DF64  cmpwi r3,0; mfmsr r4; ...;      mtmsr r5    ; OSRestoreInterrupts

cpu->msr is written from outside generated code (the runtime vectoring an
exception, or ppc_rfi). If mfmsr reads an SSA-promoted copy hoisted at region
entry, the following mtmsr writes the stale value straight back and clobbers
MSR[EE]. Same bug class as the FP-available cache and the timebase.

Fix attempted, and its result

Branch fix/msr-in-memory: slotInMemory() returns true for DOLIR_STATE_MSR
unconditionally, so MSR always lives in CPUState and is never SSA-promoted.
Module regenerated (hash 94f97ee7).

It does not fix Colosseum. Three runs: 0.4126, 0.2487, 0.2487 fps. The first
reading was an outlier; the reproducible value is 0.2487, identical to before
the change. blocked_ee essentially unchanged (1.013M vs 1.058M).

The divergence signature did change -- the EE-only pattern is no longer dominant;
a N=0x3932,I=0x1030 pattern (native has FP/FE0/FE1/RI set where the interpreter
is in exception-entry state) now accounts for 42011 of 42360. Whether that is a
genuine second bug or an artifact of the shadow interpreter is not established.

Caveat on lockstep as evidence

A Mario Kart lockstep run crashed with Invalid read from 0x00000054 at
PC=0x801a1bc8 and threw modal dialogs. The shadow interpreter re-executes
blocks and can double-issue side effects, so lockstep output in this region is
suggestive, not authoritative. (Harness scripts now write
UsePanicHandlers = False so runs cannot raise blocking dialogs.)

[dougchansan]

Exception histogram by vector (new, and the sharpest result so far)

Added a per-vector histogram in ppc_take_exception (GXRuntime/src/core/cpu_exception.c)
and regenerated both Colosseum modules. Same scene, same savestate:

arm	total	vec 0x800 (FP unavailable)	vec 0xC00 (system call)
C	19,858	9,297	10,561
LLVM	< 32	--	--

The C arm raises ~20k guest exceptions of exactly two kinds. The LLVM arm raises
fewer than 32 in total -- the histogram never even reached a dump threshold of 32.
Both sc and FP-unavailable are correctly lowered in the LLVM backend
(src/backend/llvm/control_flow.cpp:202 calls ppc_system_call_exception;
emitFPAvailable in special_instructions.cpp), so this is not missing codegen --
the guest simply never reaches that code.

Consistent with this, the one non-EE lockstep msr pattern is N=0x3932, I=0x1030:
native has MSR[FP] (0x2000) set where the interpreter has it clear. A guest with
FP permanently enabled never traps, so the OS lazy-FPU context switch never engages.
Note the FP-available cache fix did not change this, so the suspicion is MSR[FP]
itself being wrong in native state rather than the cached answer.

Thread scheduling is identical

Tracing OSCurrentThread (0x800000E4) transitions from the run loop, both arms
alternate between exactly the same two threads -- 803FBCB0 (the blocked one) and
803FB998 -- in the same pattern. The LLVM arm just switches ~8x less often. No
thread is starved out of the run queue; the structure is the same.

The framerate is bimodal