diff --git a/plans/pd_vm_call_overhead_reduction_plan.md b/plans/pd_vm_call_overhead_reduction_plan.md index 516722b1..9a6f2a45 100644 --- a/plans/pd_vm_call_overhead_reduction_plan.md +++ b/plans/pd_vm_call_overhead_reduction_plan.md @@ -11,9 +11,9 @@ separate costs in dependency order: 4. lower the hottest builtin projection calls into direct interpreter fast paths instead of paying full generic `Call` dispatch -The intent is not to redesign ordinary RustScript function calls. Those are already compiled -inline today in [`src/compiler/codegen.rs`](../src/compiler/codegen.rs), so the main -runtime tax is on builtin and host dispatch. +The first four stages did not redesign ordinary RustScript function calls. Current script functions +use real callable frames and dynamic calls route through `CallValue`. The native trace-graph +follow-up preserves those semantics and targets the remaining trace-boundary overhead. ## Current Snapshot (2026-03-20 Baseline) @@ -145,10 +145,458 @@ Interpretation: - Stage 4 remains in the expected post-fast-path regime where interpreter and JIT are close enough that specialized-trace coverage, not simple wall-clock ordering, is the stable regression guard +## Validation Rerun (2026-07-18) + +This rerun preserves the 2026-03-21 results above and compares the plan-validation revision with +the tested revision on the same machine: + +- plan-validation baseline: `4dff448b23585e1bec33b913aad781dcdc326fd1` +- tested revision (checked-out `master` at benchmark time): `73fb7879907297d47d059d4da646f129600d3d62` +- toolchain: `rustc 1.94.1 (e408947bf 2026-03-25)` +- main affinity: CPU 3; cross-check affinity: CPU 11 +- baseline and current used independent debug and release target directories +- rename-aware comparison found only formatting/repository-layout changes in the perf harness; + iteration counts, warmups, expected values, and measurement logic remain equivalent +- all 178 captured benchmark invocations passed their correctness assertions; array/map runs also + retained the expected `jit_traces=5` and `jit_traces=6` coverage + +The exact plan commands were run first. The debug host/proc-macro timings were strongly bimodal on +this VM and did not reproduce the historical absolute `0.9-1.2 us/call` host range for either +revision. Supplementary release A/B runs therefore used alternating endpoint order and a second CPU. +The collection benchmarks were repeated in release mode with the original Cranelift JIT feature. + +Same-machine A/B signals: + +| Measurement | Baseline | Current | Change | Confidence | +|---|---:|---:|---:|---| +| host `0arg` family, CPU 11 median-ratio geometric mean | - | - | `+50.0%` | regression signal; bimodal samples overlap | +| host `1arg` family, CPU 11 median-ratio geometric mean | - | - | `+24.3%` | regression signal; ABI variants are mixed | +| proc-macro family, CPU 11 median-ratio geometric mean | - | - | `+14.2%` | no aggregate improvement; bimodal samples overlap | +| no-host tight-loop interpreter median | `209 ms` | `214 ms` | `+2.4%` | close | +| array interpreter, cross-core low-latency sample | `16.54 ms` | `18.79 ms` | `+13.6%` | modest regression signal | +| array JIT, cross-core low-latency sample | `20.21 ms` | `21.20 ms` | `+4.9%` | close | +| map interpreter, cross-core low-latency sample | `18.34 ms` | `18.72 ms` | `+2.1%` | close | +| map JIT, CPU 3 low-latency sample | `22.29 ms` | `56.32 ms` | `+152.7%` | inconclusive across CPUs | +| array JIT trace count | `5` | `5` | unchanged | structural coverage retained | +| map JIT trace count | `6` | `6` | unchanged | structural coverage retained | + +The historical collection values remain reproducible by the baseline revision on the current +machine: + +| Measurement | 2026-03-21 document value | 2026-07-18 baseline rerun | +|---|---:|---:| +| array interpreter | `15.747 ms` | `16.543 ms` | +| array JIT | `16.425 ms` | `20.213 ms` | +| map interpreter | `19.582 ms` | `18.340 ms` | +| map JIT | `20.165 ms` | `22.285 ms` | + +Interpretation: + +- tested revision `73fb787` does not demonstrate a broad call-overhead improvement over `4dff448` +- the host result is call-path-specific rather than a general interpreter-loop shift: the no-host + tight-loop control moved only `+2.4%` +- dynamic `Args*` and `Stack*` `1arg` variants improved in the CPU 11 median comparison, while + static, legacy, and most `0arg` variants produced the aggregate host regression signal +- proc-macro heap-argument paths show no repeatable aggregate gain +- array/map interpreter paths remain close to baseline, with array showing a modest slowdown signal +- map JIT retained six traces and passed all specialized-op coverage assertions; CPU 3 did not recover + the baseline low-latency cluster, but the large delta did not reproduce consistently on CPU 11, so + the wall-clock result remains inconclusive rather than a confirmed regression +- performance counters were unavailable because `perf_event_paranoid=4`; confirming any noisy CPU + regression requires a same-frequency cluster across both endpoints, at least 15 alternating rounds, + or a repeatable commit boundary from an isolated bisection +- `7a151fc` added typed builtin fast-path checks and counters to the interpreter `Call` path after the + plan baseline; it is a profiling candidate, not an established causal commit + +### `script-bench-rs` callable-frame regression and mitigation (2026-07-18) + +A separate same-machine rerun used `/mnt/TEMP/workspace/script-bench-rs` at +`6fd2a61f0a2164c40f841d585ac7dbae55362b15` and its `Sort Rust objects` workload over 10,000 +Rust-backed objects. The baseline used crates.io `pd-vm 0.23.1`; the candidate used the RustScript +working tree based on `2e5399c` with the benchmark dependency redirected by path in an isolated +export. The benchmark repository itself was not modified or pushed. + +The initial CPU 11 result, four alternating Criterion runs with `sample_size=10`, was: + +| Mode | `0.23.1` median | `master` median before fix | Change | +|---|---:|---:|---:| +| interpreter | `330.6 ms` | `536.6 ms` | `+62.3%` | +| JIT | `81.1 ms` | `1875.6 ms` | `+2212.4%` / `23.1x` slower | + +CPU 3 reproduced the JIT delta at `23.3x`, so the result was not a scheduler-frequency artifact. +Native-trace assertions passed on both endpoints. + +Bisection and trace diagnostics found: + +- first large JIT regression: `df6a0ff feat(jit): trace inside script callable frames` +- all hot sort traces ran in prototype frame `0` +- one reused pre-fix run produced about `352k` native executions, `352k` exits, `0` native + loopbacks, and `332k` linked handoffs +- trace shape and execution counts were close to `0.23.1`; the regression came from repeatedly + entering and leaving frame-relative native traces that performed no sustained native loopback +- after a trace was blocked, the interpreter still eagerly built `active_local_types()` before the + engine could reject the frame, keeping the JIT-enabled fallback path materially slower + +The mitigation is callable-frame adaptive backoff: + +- count consecutive non-loopback native side exits for callable-frame traces +- reset the streak after a successful native loopback +- after 64 consecutive side exits, block the complete callable prototype frame and remove its + compiled entry mappings +- check the blocked-frame bitmap before constructing frame-local type vectors +- root traces remain eligible, and scalar callable loops with native loopbacks remain native + +Post-fix CPU 11 A/B, four alternating runs: + +| Mode | Median | Range | +|---|---:|---:| +| current interpreter | `510.6 ms` | `490.6-537.0 ms` | +| fixed JIT | `530.3 ms` | `485.3-551.0 ms` | + +The fixed JIT is `71.7%` faster than the pre-fix master JIT and is within `+3.9%` of the same-run +interpreter median. It remains `6.54x` slower than the old `0.23.1` JIT. The old compiler flattened +direct named calls, while the current runtime preserves real callable frames and routes dynamic +calls through `CallValue`. Source-level direct-call flattening is retained only as a historical +performance reference. The follow-up design must preserve callable semantics and remove native +trace-boundary overhead inside the JIT. + +### Follow-up native trace graph plan (2026-07-18) + +#### Research basis + +LuaJIT v2.1 was used as the architectural reference for the remaining gap: + +- hot loops become root traces; the default `hotloop` threshold is `56` +- a repeatedly taken guard exit starts a side trace at the default `hotexit=10` +- the side trace is initialized by replaying the parent exit snapshot +- `lj_asm_patchexit()` redirects the parent guard to the side-trace machine-code entry +- the side-trace head inherits parent register and spill state instead of reconstructing the full + interpreter state +- Lua calls and returns can be recorded in one trace; the call target is guarded by closure or + prototype identity, while frame state remains represented in trace slots and snapshots +- actual interpreter restoration is reserved for an unlinked exit, failed guard path, error, or + other true deoptimization +- compile failures are managed per exit with `tryside` and `maxside` limits instead of disabling an + entire callable prototype after one frequent branch pattern + +A local LuaJIT experiment with a tight loop, a Lua comparator call, and a two-way branch produced +one root trace and two side traces: + +```text +[TRACE 1 lj-tight-loop.lua:8 loop] +[TRACE 2 (1/5) lj-tight-loop.lua:2 -> 1] +[TRACE 3 (1/3) lj-tight-loop.lua:7 -> 1] +``` + +The root recording contained `CALL`, `FUNCF`, comparator bytecode, `RET1`, and `FORL`. Both hot side +traces linked back to the root trace without interpreter dispatch. + +RustScript currently performs a materially different transition for every linked trace: + +1. materialize dirty locals and update `Vm.ip` +2. call `resume_linked_trace` +3. look up the next entry in Rust by `(frame_key, ip, stack_depth)` +4. invoke the next native function +5. call `pd_vm_native_frame_state()` again at callable-frame entry + +The unbacked callable JIT produced roughly `359k-392k` native executions per sort run. The three +CPU 11 diagnostic runs amortized to `4.2-7.3 us` per native boundary plus trace body, with a +`6.6 us` median. Removing a single frame helper cannot close the gap while these boundaries remain. + +#### Shared correctness constraints + +Both directions below must preserve these invariants: + +- source-level function values, closures, recursion, captures, and dynamic call targets keep their + existing semantics +- a linked native edge cannot bypass a yield, wait, fuel, epoch, error, or debugger boundary +- values transferred across a linked edge retain the existing drop contract exactly once +- an unlinked or invalidated edge uses the existing deoptimization path and reconstructs the same + `Vm.stack`, `Vm.locals`, `Vm.ip`, and active frame +- linked code must not keep stale callable, trace, executable-buffer, or captured-value identity +- trace invalidation disconnects incoming edges before executable memory or constants are released +- initial rollout links only traces with the same `frame_key`, stack depth, and non-yielding status +- root and callable traces retain independent entry keys and debug metadata +- interruption checks remain reachable in native loops after edges are linked + +#### Direction 1: fuse a hot trace region into one Cranelift function + +**Objective:** Compile a root trace and its hot same-frame side traces as one native control-flow +region so linked edges pass SSA values directly between Cranelift blocks. + +**Why first:** This removes the dominant handoff without introducing a cross-buffer ABI, writable +machine-code patching, or a tail-call convention. It also provides the parent-exit lineage and +snapshot-import model that Direction 2 needs later. + +**Target architecture:** + +- identify links by `TraceExitKey { parent_trace_id, exit_id }`, not only the destination + `(frame_key, ip, stack_depth)` +- count and compile hot exits independently +- represent a region as one root trace plus a bounded set of side traces +- import each side trace from the exact parent `SsaExit` materializations +- map the parent exit's live locals and operand values to child block parameters +- lower a linked parent guard to an intra-function Cranelift branch +- keep scalar and pointer values in SSA form across the edge +- retain the original deopt block for every unlinked outgoing edge +- run the callable frame-state entry guard once at the external region entry; internal side-trace + blocks do not call it again +- recompile and atomically replace a region entry when a newly hot side trace is admitted +- cap region growth by trace count, IR instruction count, machine-code size, and compile latency + +**Primary files:** + +- `src/vm/jit/trace.rs` + - add parent-exit hotness, link state, compile-attempt state, and region membership +- `src/vm/jit/ir.rs` + - add region-level block/value identity and parent-snapshot import metadata +- `src/vm/jit/deopt.rs` + - expose exact parent-exit materializations for child import +- `src/vm/jit/native/lower.rs` + - lower multiple SSA traces into one Cranelift function and branch directly across linked edges +- `src/vm/jit/native/mod.rs` + - represent a compiled region and its entry/keepalive state +- `src/vm/jit/runtime.rs` + - install region replacements and leave Rust dispatch only for unlinked exits +- `tests/jit/jit_tests.rs` + - add structural, callable-frame, deopt, interruption, and ownership tests +- `tests/jit/perf_tests.rs` + - add region-link characterization counters + +**Implementation milestones:** + +1. **Add parent-exit identity and counters.** + - RED: a branch-heavy callable workload must report repeated executions of one exact parent exit. + - GREEN: record per-exit executions without changing dispatch. + - Keep the existing frame backoff active during this milestone. + +2. **Create side-trace snapshot import IR.** + - RED: importing a child from a parent exit must preserve scalar, tagged, stack, and dirty-local + materializations. + - GREEN: build child entry parameters from `SsaExit` values and verify the combined IR. + - Reject cross-frame, yielding, mismatched-depth, and unsupported ownership edges. + +3. **Lower a two-trace region.** + - RED: a hot alternating branch must still increment Rust linked-handoff counters after warmup. + - GREEN: compile parent and one child in the same Cranelift function; the linked edge becomes an + internal branch and no longer increments the handoff counter. + - Preserve the original parent deopt block until the replacement region is installed. + +4. **Support cycles and several side traces.** + - RED: a parent-to-child-to-parent loop must require repeated external native entries. + - GREEN: map the cycle to Cranelift blocks and PHI-style block parameters with no host-stack + growth. + - Add deterministic region-size limits and refuse an edge before partial installation. + +5. **Install and invalidate regions safely.** + - RED: JIT config changes, host-import changes, trace blocking, and callable replacement must not + leave an incoming edge pointing at old code. + - GREEN: publish a complete replacement region, switch the cached entry, then retire the prior + keepalive after all references are disconnected. + +6. **Retire prototype-wide backoff for successfully linked exits.** + - Keep backoff for unsupported or repeatedly failing exits. + - Move compile attempts and blacklist state to `TraceExitKey`. + - A successful region edge clears its failure streak and cannot block unrelated exits in the + same callable frame. + +7. **Run the sort performance gate.** + - use CPU 11 with four alternating Criterion runs + - compare current interpreter, region JIT, and crates.io `0.23.1` + - record native executions, external exits, internal region edges, Rust handoffs, helper calls, + trace count, region count, IR size, machine-code size, and compile time + +**Required tests:** + +- `trace_jit_region_links_hot_same_frame_side_exit` +- `trace_jit_region_cycles_without_external_handoffs` +- `trace_jit_region_unlinked_exit_restores_callable_frame` +- `trace_jit_region_preserves_owned_value_drop_contract` +- `trace_jit_region_respects_fuel_and_epoch_interrupts` +- `trace_jit_region_invalidation_disconnects_old_entries` +- `trace_jit_region_rejects_cross_frame_or_yielding_edges` + +**Acceptance criteria:** + +- all existing workspace tests and clippy remain clean +- fused edges perform no Rust lookup, frame-state helper call, or `Vm.locals` materialization +- the branch-heavy callable integration test shows internal region-edge activity and bounded external + handoffs on its second run +- `script-bench-rs` preserves sorted output and native coverage +- the sort JIT median is at most `2x` the `0.23.1` JIT median on the same CPU and at least `50%` + below the same-run interpreter median +- if the performance gate is missed, record a helper/edge/call breakdown before widening the region + or starting Direction 2 + +#### Direction 2: independent side traces with machine-level links + +**Objective:** Keep side traces in independent executable buffers while linking parent exits to +child entries without Rust dispatch or full VM-state reconstruction. + +**When to use:** Start after Direction 1 has validated parent-exit lineage and snapshot import, and +only if region recompilation, code duplication, or region-size limits prevent adequate coverage. + +**Target architecture:** + +- each hot parent exit owns a stable link slot outside executable memory +- the parent guard loads the link slot and transfers control to the child when populated +- a zero or invalidated slot follows the original deopt path +- side traces are compiled from one exact parent snapshot and have one inherited-state ABI +- a generated system-ABI wrapper enters the trace graph +- linked trace bodies use Cranelift's tail-call convention and `return_call_indirect` so graph cycles + do not grow the host stack +- scalar values cross the side-entry ABI unboxed +- tagged and owned values use an ABI-safe carrier with explicit move/borrow ownership; pointers to a + parent function's temporary stack slots cannot survive a tail transfer +- child code can tail-link to the root or another compatible side trace +- link publication uses release ordering; execution loads use acquire ordering +- invalidation clears incoming link slots before releasing child code and constants +- executable pages remain write-protected after publication; mutable link slots avoid runtime + machine-code rewriting and preserve current W^X behavior + +**Cranelift constraints discovered during research:** + +- the current dependency is `cranelift-codegen 0.129.1` +- `return_call_indirect` is available +- only `CallConv::Tail` reports tail-call support; the current default platform convention does not +- current traces are compiled independently and copied into separate `ExecutableBuffer` mappings +- current `ExecutableBuffer` exposes no runtime patch API +- a system-ABI wrapper plus tail-call trace bodies and data link slots is therefore preferred over + editing executable bytes + +**Primary files:** + +- all Direction 1 trace-lineage, snapshot, and test surfaces +- `src/vm/jit/native/lower.rs` + - add side-entry signatures, tail transfers, and link-slot loads +- `src/vm/jit/native/mod.rs` + - model root wrappers, tail-call bodies, link slots, and keepalive dependencies +- `src/vm/native/exec.rs` + - retain W^X mappings; add patch support only if data slots prove insufficient +- `src/vm/jit/runtime.rs` + - publish/unpublish links and service only unlinked deoptimizations + +**Implementation milestones:** + +1. **Define and verify inherited-state ABI classes.** + - classify scalar, pointer, tagged, borrowed, and owned values + - prove exact ownership transfer and deopt reconstruction for every class + - reject unsupported signatures before native code is published + +2. **Build a root wrapper and one tail-linked child.** + - system ABI enters the wrapper + - wrapper calls the `CallConv::Tail` root body + - root uses `return_call_indirect` for the populated side link + - test the zero-slot deopt path and populated-slot direct path + +3. **Add cyclic graph linking.** + - parent, child, and root may tail-link without host-stack growth + - run millions of transitions under a stack-depth probe + - ensure interruption and error exits return through the wrapper correctly + +4. **Add link lifetime and invalidation.** + - track incoming links for each compiled trace + - clear all incoming slots before code retirement + - prevent ABA reuse with generation identity or non-reused link objects + - keep child constants and executable memory alive while any slot can reach them + +5. **Replace runtime dispatch for compatible edges.** + - keep Rust dispatch for cold, unsupported, yielding, cross-frame, and invalidated exits + - collect separate direct-link and fallback-handoff counters + +6. **Run the same correctness and performance gates as Direction 1.** + - compare direct-linked side traces with fused regions on identical trace graphs + - prefer the simpler fused region unless independent traces provide a measured compile-time, + code-size, or coverage advantage + +**Required tests:** + +- `trace_jit_side_link_slot_switches_between_deopt_and_child` +- `trace_jit_tail_link_cycle_has_bounded_host_stack` +- `trace_jit_side_entry_transfers_owned_values_once` +- `trace_jit_side_link_invalidation_clears_incoming_slots` +- `trace_jit_side_link_generation_prevents_stale_entry_reuse` +- `trace_jit_side_link_respects_callable_frame_and_interrupt_boundaries` + +**Acceptance criteria:** + +- compatible linked exits execute with no Rust dispatcher or callable frame-state helper +- graph cycles show bounded host-stack use +- no executable page is writable while reachable for execution +- clearing a link immediately restores the existing deopt behavior +- owned and captured values retain exact release behavior under linking and invalidation +- performance is no worse than Direction 1 on the sort workload; any added complexity must be + justified by measured region compile-time, code-size, or coverage gains + +**Implemented inherited-state checkpoint (2026-07-19):** + +- independent trace dispatchers now use a two-argument tail ABI `(vm, inherited_state)` beneath a + one-argument system-ABI wrapper +- the wrapper owns a bounded packet containing frame identity, active stack/local bases, canonical + stack/local `Value` addresses, and a dynamic cross-frame continuation target +- same-frame linked exits restore their classified state once, publish packet addresses, and tail to + the child without a child `frame_state` bridge call +- native `CallValue` and return transitions populate the same packet after changing frames; the + dispatcher may tail to the already-compiled trace selected for the resulting active frame +- frame-key validation rejects stale or incompatible packets and falls back to the original guarded + VM reconstruction path +- default direct publication is enabled for same-frame edges; static cross-frame slots remain + opt-in, while call/return transitions use the frame-derived dynamic target to avoid polymorphic + continuation corruption +- direct progress no longer activates callable-wide backoff or the old global region fallback; + unsupported, cold, yielding, invalidated, and missing-target paths still return to Rust +- focused direct-cycle telemetry requires at most four steady-state `frame_state` bridge hits; the + escaped callable fixture retains bounded bridge reloads and all direct-link lifetime, generation, + interruption, ownership, and stack-depth tests remain active + +CPU 11 four-round alternating Criterion gate, frozen crates.io `pd-vm 0.23.1` binary versus the +inherited-state candidate (point estimates, milliseconds): + +| Round | `0.23.1` | inherited-state candidate | +|---:|---:|---:| +| 1 | `61.289` | `132.890` | +| 2 | `82.893` | `188.570` | +| 3 | `88.002` | `118.330` | +| 4 | `46.631` | `136.100` | +| median | `72.091` | `134.495` | + +The candidate/base median ratio is `1.866x`, satisfying the `<=2x` sort gate. The candidate median +is `67.95%` below the pre-inherited-state `419.614 ms` median. Correct sorted output and native +execution assertions passed in every Criterion run. Raw logs were captured under +`/tmp/inherited-state-results/final-ab-{base,candidate}-*.log` during validation; they are temporary +session artifacts and are not part of the repository. + +#### Direction selection and shared rollout + +Direction 1 remains the fused-region fallback and comparison baseline. Direction 2 is now the default +same-frame linking mode after satisfying the shared correctness and performance gates; static +cross-frame slots stay opt-in, while frame-derived dynamic call/return targets use the same packet +without making a polymorphic continuation slot global. The following components remain shared by +both directions: + +- `TraceExitKey` and per-exit hotness/attempt/blacklist state +- parent snapshot import and inherited-value classification +- linked-edge counters and diagnostics +- deopt equivalence tests +- callable-frame, ownership, interruption, and invalidation test fixtures +- region/side-trace size and compile-time telemetry + +Source compiler direct-call flattening is not part of either direction. A later call-tracing phase may +record guarded `CallValue -> callee -> return` paths into the native trace graph while preserving real +frames in snapshots. + +Verification: + +- focused trace-engine, callable-frame, and stale-continuation regression tests passed +- complete `jit_tests`: `128 passed`, `0 failed`, `11 ignored` +- `RUSTFLAGS="-Dwarnings" cargo clippy --workspace --all-targets --all-features` passed +- `cargo test --workspace` passed +- GitHub CI passed on Ubuntu, macOS, and Windows + ## Non-Goals -- redesigning RustScript source-level function semantics -- introducing general closure values or call frames as part of this work +- changing RustScript source-level function, closure, capture, or recursion semantics +- restoring source-compiler function flattening or removing real callable frames - removing `Arc` from the VM value model - broad AOT architecture work beyond the minimal compatibility needed for the hot builtin lowering - changing user-visible RSS syntax as part of this plan diff --git a/src/lib.rs b/src/lib.rs index bf9b7113..833ad637 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -9,8 +9,8 @@ pub mod debugger; #[cfg(feature = "runtime")] pub mod jit { pub use crate::vm::jit::{ - JitAttempt, JitConfig, JitMetrics, JitNyiDoc, JitNyiReason, JitSnapshot, JitTrace, - JitTraceTerminal, TraceJitEngine, + JitAttempt, JitConfig, JitExitProfile, JitMetrics, JitNyiDoc, JitNyiReason, JitSnapshot, + JitTrace, JitTraceTerminal, TraceJitEngine, }; } #[cfg(feature = "cli")] @@ -81,8 +81,8 @@ pub use debugger::{ }; #[cfg(feature = "runtime")] pub use jit::{ - JitAttempt, JitConfig, JitMetrics, JitNyiDoc, JitNyiReason, JitSnapshot, JitTrace, - JitTraceTerminal, TraceJitEngine, + JitAttempt, JitConfig, JitExitProfile, JitMetrics, JitNyiDoc, JitNyiReason, JitSnapshot, + JitTrace, JitTraceTerminal, TraceJitEngine, }; #[cfg(feature = "runtime")] pub use vm::diagnostics::render_vm_error; diff --git a/src/vm/host.rs b/src/vm/host.rs index ef3381ff..83f9ee4b 100644 --- a/src/vm/host.rs +++ b/src/vm/host.rs @@ -313,9 +313,10 @@ impl HostFunctionRegistry { /// Registers a static args-only host function that always returns one value synchronously. /// - /// Returning no value, `Halt`, `Yield`, or `Pending` violates the contract and is reported as - /// a host error. When appropriate, the native JIT may keep traces active across the call - /// boundary. + /// The returned [`Value`] must match the return type declared by the corresponding host + /// import. Returning a different type is reported as [`VmError::TypeMismatch`]. Returning no + /// value, `Halt`, `Yield`, or `Pending` violates the contract and is reported as a host error. + /// When appropriate, the native JIT may keep traces active across the call boundary. pub fn register_static_non_yielding_args( &mut self, name: impl Into, @@ -494,6 +495,41 @@ pub(crate) fn require_non_yielding_host_value(outcome: CallOutcome) -> VmResult< } } +pub(crate) fn validate_non_yielding_host_value( + value: Value, + expected: Option, +) -> VmResult { + let valid = matches!( + (expected, &value), + (None | Some(ValueType::Unknown), _) + | (Some(ValueType::Null), Value::Null) + | (Some(ValueType::Int), Value::Int(_)) + | (Some(ValueType::Float), Value::Float(_)) + | (Some(ValueType::Bool), Value::Bool(_)) + | (Some(ValueType::String), Value::String(_)) + | (Some(ValueType::Bytes), Value::Bytes(_)) + | (Some(ValueType::Array), Value::Array(_)) + | (Some(ValueType::Map), Value::Map(_)) + | (Some(ValueType::Callable), Value::Callable(_)) + ); + if valid { + return Ok(value); + } + let expected = match expected.expect("known expected host return type") { + ValueType::Unknown => unreachable!(), + ValueType::Null => "null", + ValueType::Int => "int", + ValueType::Float => "float", + ValueType::Bool => "bool", + ValueType::String => "string", + ValueType::Bytes => "bytes", + ValueType::Array => "array", + ValueType::Map => "map", + ValueType::Callable => "callable", + }; + Err(VmError::TypeMismatch(expected)) +} + #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub(super) struct WaitingHostOp { pub(super) op_id: HostOpId, @@ -573,7 +609,10 @@ impl Vm { /// Registers a static args-only host function that always returns one value synchronously. /// - /// Returning no value, `Halt`, `Yield`, or `Pending` violates the contract and is a host error. + /// When used to resolve a declared host import, the returned [`Value`] must match that + /// import's return type. Returning a different type is reported as + /// [`VmError::TypeMismatch`]. Returning no value, `Halt`, `Yield`, or `Pending` violates the + /// contract and is a host error. pub fn register_static_non_yielding_args_function( &mut self, function: StaticHostArgsFunction, @@ -723,8 +762,9 @@ impl Vm { /// Binds a static args-only host function that always returns one value synchronously. /// /// This is equivalent to [`Vm::bind_static_args_function`] except that the VM may keep - /// native JIT traces active across the call boundary. Returning no value, `Halt`, `Yield`, - /// or `Pending` violates the contract and is reported as a host error. + /// native JIT traces active across the call boundary. The returned [`Value`] must match the + /// return type declared by the corresponding host import. Returning a different type, no + /// value, `Halt`, `Yield`, or `Pending` violates the contract and is reported as a host error. pub fn bind_static_non_yielding_args_function( &mut self, name: impl Into, @@ -948,9 +988,19 @@ impl Vm { return self.execute_builtin_call_from_stack(builtin, argc, call_ip); } + let expected_return_type = self + .program + .imports + .get(usize::from(index)) + .map(|import| import.return_type); let resolved_index = self.resolve_call_target(index, argc_u8)?; if self.bound_host_function_uses_args_slice(resolved_index)? { - self.execute_bound_args_host_function(resolved_index, argc, call_ip) + self.execute_bound_args_host_function( + resolved_index, + argc, + call_ip, + expected_return_type, + ) } else if self.bound_host_function_uses_stack_borrow(resolved_index)? { self.execute_bound_stack_host_function(resolved_index, argc, call_ip) } else { @@ -975,7 +1025,7 @@ impl Vm { })?; let argc = argc_u8 as usize; if self.bound_host_function_uses_args_slice(resolved_index)? { - self.execute_bound_args_host_function(resolved_index, argc, call_ip) + self.execute_bound_args_host_function(resolved_index, argc, call_ip, None) } else if self.bound_host_function_uses_stack_borrow(resolved_index)? { self.execute_bound_stack_host_function(resolved_index, argc, call_ip) } else { @@ -1492,6 +1542,7 @@ impl Vm { resolved_index: u16, argc: usize, call_ip: usize, + expected_return_type: Option, ) -> VmResult { let arg_start = self .stack @@ -1520,6 +1571,7 @@ impl Vm { let outcome = outcome?; if non_yielding { let value = require_non_yielding_host_value(outcome)?; + let value = validate_non_yielding_host_value(value, expected_return_type)?; self.stack.truncate(arg_start); self.stack.push(value); return Ok(HostCallExecOutcome::Returned); diff --git a/src/vm/jit/deopt.rs b/src/vm/jit/deopt.rs index 4d3c67c7..d28cd88b 100644 --- a/src/vm/jit/deopt.rs +++ b/src/vm/jit/deopt.rs @@ -1,5 +1,7 @@ #![allow(dead_code)] -use super::ir::{SsaExit, SsaMaterialization, SsaValue, SsaValueId, SsaValueRepr}; +use super::ir::{ + SsaExit, SsaExitId, SsaMaterialization, SsaTrace, SsaValue, SsaValueId, SsaValueRepr, +}; pub(crate) fn materialize_ssa_values( values: impl IntoIterator, @@ -41,3 +43,211 @@ pub(crate) fn exit_inputs(exit: &SsaExit) -> Vec { } out } + +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct SideTraceImport { + pub(crate) parent_exit: SsaExitId, + pub(crate) stack_depth: usize, + pub(crate) local_count: usize, + pub(crate) dirty_locals: Vec, + pub(crate) args: Vec, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum SideTraceImportError { + UnknownParentExit(SsaExitId), + ExitIpMismatch { parent: usize, child: usize }, + StackDepthMismatch { parent: usize, child: usize }, + LocalCountMismatch { parent: usize, child: usize }, + InvalidChildEntry, +} + +pub(crate) fn side_trace_import( + parent: &SsaTrace, + parent_exit: SsaExitId, + child: &SsaTrace, +) -> Result { + let exit = parent + .exits + .iter() + .find(|exit| exit.id == parent_exit) + .ok_or(SideTraceImportError::UnknownParentExit(parent_exit))?; + if exit.exit_ip != child.root_ip { + return Err(SideTraceImportError::ExitIpMismatch { + parent: exit.exit_ip, + child: child.root_ip, + }); + } + if exit.stack.len() != child.entry_stack_depth { + return Err(SideTraceImportError::StackDepthMismatch { + parent: exit.stack.len(), + child: child.entry_stack_depth, + }); + } + let child_entry = child + .blocks + .get(child.entry.index()) + .ok_or(SideTraceImportError::InvalidChildEntry)?; + let child_local_count = child_entry + .params + .len() + .checked_sub(child.entry_stack_depth) + .ok_or(SideTraceImportError::InvalidChildEntry)?; + if exit.locals.len() != child_local_count { + return Err(SideTraceImportError::LocalCountMismatch { + parent: exit.locals.len(), + child: child_local_count, + }); + } + + Ok(SideTraceImport { + parent_exit, + stack_depth: exit.stack.len(), + local_count: exit.locals.len(), + dirty_locals: exit.dirty_locals.clone(), + args: exit.stack.iter().chain(&exit.locals).cloned().collect(), + }) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::vm::jit::ir::{SsaTerminator, SsaTraceBuilder, SsaValueRepr}; + + #[test] + fn side_trace_import_maps_parent_stack_then_locals_to_child_entry() { + let mut parent = SsaTraceBuilder::new(0, 1); + let parent_entry = parent.entry(); + let stack = parent + .append_param(parent_entry, SsaValueRepr::Tagged, "stack0".to_string()) + .unwrap(); + let local = parent + .append_param(parent_entry, SsaValueRepr::I64, "local0".to_string()) + .unwrap(); + let exit_id = parent.add_exit( + 12, + vec![SsaMaterialization::Value(stack.id)], + vec![SsaMaterialization::BoxInt(local.id)], + vec![true], + ); + parent + .set_terminator(parent_entry, SsaTerminator::Exit { exit: exit_id }) + .unwrap(); + let parent = parent.finish(); + + let mut child = SsaTraceBuilder::new(12, 1); + let child_entry = child.entry(); + child + .append_param(child_entry, SsaValueRepr::Tagged, "stack0".to_string()) + .unwrap(); + child + .append_param(child_entry, SsaValueRepr::Tagged, "local0".to_string()) + .unwrap(); + let child_exit = child.add_exit(13, Vec::new(), Vec::new(), Vec::new()); + child + .set_terminator(child_entry, SsaTerminator::Exit { exit: child_exit }) + .unwrap(); + let child = child.finish(); + + let import = side_trace_import(&parent, exit_id, &child).unwrap(); + + assert_eq!(import.parent_exit, exit_id); + assert_eq!(import.stack_depth, 1); + assert_eq!(import.local_count, 1); + assert_eq!(import.dirty_locals, vec![true]); + assert_eq!( + import.args, + vec![ + SsaMaterialization::Value(stack.id), + SsaMaterialization::BoxInt(local.id), + ] + ); + } + + #[test] + fn side_trace_import_rejects_mismatched_exit_ip() { + let mut parent = SsaTraceBuilder::new(0, 0); + let parent_entry = parent.entry(); + let exit_id = parent.add_exit(12, Vec::new(), Vec::new(), Vec::new()); + parent + .set_terminator(parent_entry, SsaTerminator::Exit { exit: exit_id }) + .unwrap(); + let parent = parent.finish(); + + let mut child = SsaTraceBuilder::new(13, 0); + let child_entry = child.entry(); + let child_exit = child.add_exit(14, Vec::new(), Vec::new(), Vec::new()); + child + .set_terminator(child_entry, SsaTerminator::Exit { exit: child_exit }) + .unwrap(); + let child = child.finish(); + + assert_eq!( + side_trace_import(&parent, exit_id, &child), + Err(SideTraceImportError::ExitIpMismatch { + parent: 12, + child: 13, + }) + ); + } + + #[test] + fn side_trace_import_rejects_mismatched_stack_depth() { + let mut parent = SsaTraceBuilder::new(0, 0); + let parent_entry = parent.entry(); + let exit_id = parent.add_exit(12, Vec::new(), Vec::new(), Vec::new()); + parent + .set_terminator(parent_entry, SsaTerminator::Exit { exit: exit_id }) + .unwrap(); + let parent = parent.finish(); + + let mut child = SsaTraceBuilder::new(12, 1); + let child_entry = child.entry(); + child + .append_param(child_entry, SsaValueRepr::Tagged, "stack0".to_string()) + .unwrap(); + let child_exit = child.add_exit(13, Vec::new(), Vec::new(), Vec::new()); + child + .set_terminator(child_entry, SsaTerminator::Exit { exit: child_exit }) + .unwrap(); + let child = child.finish(); + + assert_eq!( + side_trace_import(&parent, exit_id, &child), + Err(SideTraceImportError::StackDepthMismatch { + parent: 0, + child: 1, + }) + ); + } + + #[test] + fn side_trace_import_rejects_mismatched_local_count() { + let mut parent = SsaTraceBuilder::new(0, 0); + let parent_entry = parent.entry(); + let exit_id = parent.add_exit(12, Vec::new(), Vec::new(), Vec::new()); + parent + .set_terminator(parent_entry, SsaTerminator::Exit { exit: exit_id }) + .unwrap(); + let parent = parent.finish(); + + let mut child = SsaTraceBuilder::new(12, 0); + let child_entry = child.entry(); + child + .append_param(child_entry, SsaValueRepr::Tagged, "local0".to_string()) + .unwrap(); + let child_exit = child.add_exit(13, Vec::new(), Vec::new(), Vec::new()); + child + .set_terminator(child_entry, SsaTerminator::Exit { exit: child_exit }) + .unwrap(); + let child = child.finish(); + + assert_eq!( + side_trace_import(&parent, exit_id, &child), + Err(SideTraceImportError::LocalCountMismatch { + parent: 0, + child: 1, + }) + ); + } +} diff --git a/src/vm/jit/ir.rs b/src/vm/jit/ir.rs index a5341234..8f856331 100644 --- a/src/vm/jit/ir.rs +++ b/src/vm/jit/ir.rs @@ -10,6 +10,10 @@ impl SsaValueId { pub(crate) fn new(raw: u32) -> Self { Self(raw) } + + pub(crate) fn raw(self) -> u32 { + self.0 + } } impl fmt::Display for SsaValueId { @@ -44,6 +48,10 @@ impl SsaExitId { pub(crate) fn new(raw: u32) -> Self { Self(raw) } + + pub(crate) fn raw(self) -> u32 { + self.0 + } } impl fmt::Display for SsaExitId { @@ -180,9 +188,13 @@ pub(crate) enum SsaInstKind { BytesFromArrayU8 { array: SsaValueId, }, + BytesToUtf8Ascii { + bytes: SsaValueId, + }, BytesToArrayU8 { bytes: SsaValueId, }, + ArrayNew, ArrayLen { array: SsaValueId, }, @@ -420,7 +432,8 @@ impl SsaInstKind { Self::StringSplitLiteral { text, delimiter } => vec![*text, *delimiter], Self::StringConcat { lhs, rhs } | Self::BytesConcat { lhs, rhs } => vec![*lhs, *rhs], Self::BytesFromArrayU8 { array } => vec![*array], - Self::BytesToArrayU8 { bytes } => vec![*bytes], + Self::BytesToUtf8Ascii { bytes } | Self::BytesToArrayU8 { bytes } => vec![*bytes], + Self::ArrayNew => Vec::new(), Self::ArrayGet { array, index } => vec![*array, *index], Self::ArrayHas { array, index } => vec![*array, *index], Self::ArraySet { @@ -1059,7 +1072,9 @@ fn render_inst_kind(kind: &SsaInstKind) -> String { SsaInstKind::StringConcat { lhs, rhs } => format!("string_concat {lhs}, {rhs}"), SsaInstKind::BytesConcat { lhs, rhs } => format!("bytes_concat {lhs}, {rhs}"), SsaInstKind::BytesFromArrayU8 { array } => format!("bytes_from_array_u8 {array}"), + SsaInstKind::BytesToUtf8Ascii { bytes } => format!("bytes_to_utf8_ascii {bytes}"), SsaInstKind::BytesToArrayU8 { bytes } => format!("bytes_to_array_u8 {bytes}"), + SsaInstKind::ArrayNew => "array_new".to_string(), SsaInstKind::ArrayLen { array } => format!("array_len {array}"), SsaInstKind::ArrayGet { array, index } => format!("array_get {array}, {index}"), SsaInstKind::ArrayHas { array, index } => format!("array_has {array}, {index}"), diff --git a/src/vm/jit/mod.rs b/src/vm/jit/mod.rs index 006b6c68..bbae02a7 100644 --- a/src/vm/jit/mod.rs +++ b/src/vm/jit/mod.rs @@ -3,11 +3,12 @@ pub(crate) mod ir; pub(crate) mod liveness; pub(crate) mod native; pub(crate) mod recorder; +pub(crate) mod region; pub(crate) mod runtime; pub(crate) mod trace; pub(crate) use runtime::NativeTrace; pub use trace::{ - JitAttempt, JitConfig, JitMetrics, JitNyiDoc, JitNyiReason, JitSnapshot, JitTrace, - JitTraceTerminal, TraceJitEngine, + JitAttempt, JitConfig, JitExitProfile, JitMetrics, JitNyiDoc, JitNyiReason, JitSnapshot, + JitTrace, JitTraceTerminal, TraceJitEngine, }; diff --git a/src/vm/jit/native/lower.rs b/src/vm/jit/native/lower.rs index 4dd4068e..bee4b6ce 100644 --- a/src/vm/jit/native/lower.rs +++ b/src/vm/jit/native/lower.rs @@ -1,5 +1,6 @@ -use super::super::super::{Value, VmError, VmResult}; +use super::super::super::{Value, ValueType, VmError, VmResult}; use super::super::JitTrace; +use super::super::region::FusedRegionLink; use super::super::runtime::resume_linked_trace_entry_address; use super::{NativeCompileProfile, TraceLoweringKind}; use crate::vm::jit::deopt::exit_inputs; @@ -15,13 +16,16 @@ use crate::vm::native::{ array_set_signature, box_heap_value_signature, checked_add_i32, clear_value_slot_entry_address, clone_value_signature, clone_value_to_slot_entry_address, collection_get_signature, collection_predicate_signature, copy_bytes_entry_address, copy_bytes_signature, - detect_native_stack_layout, enter_call_value_entry_address, enter_call_value_signature, - entry_signature, frame_state_entry_address, frame_state_signature, free_buffer_signature, - jump_with_status, leave_frame_entry_address, leave_frame_signature, map_get_entry_address, + detect_native_stack_layout, enter_call_value_inherited_entry_address, + enter_call_value_inherited_signature, entry_signature, frame_state_entry_address, + frame_state_signature, free_buffer_signature, jump_with_status, + leave_frame_inherited_entry_address, leave_frame_inherited_signature, map_get_entry_address, map_has_entry_address, map_iter_next_entry_address, map_iter_next_signature, map_iter_take_key_entry_address, map_iter_take_signature, map_iter_take_value_entry_address, map_set_entry_address, map_set_signature, non_yielding_host_call_entry_address, - non_yielding_host_call_signature, pack_shared_signature, regex_match_entry_address, + non_yielding_host_call_signature, non_yielding_i64_host_call_entry_address, + non_yielding_i64_host_call_signature, non_yielding_scalar_host_call_entry_address, + non_yielding_scalar_host_call_signature, pack_shared_signature, regex_match_entry_address, regex_match_signature, regex_replace_entry_address, regex_replace_signature, restore_active_sparse_exit_state_entry_address, shared_array_from_buffer_entry_address, shared_bytes_from_buffer_entry_address, shared_string_from_buffer_entry_address, @@ -36,9 +40,10 @@ use crate::vm::native::{ use cranelift_codegen::ir::condcodes::{FloatCC, IntCC}; use cranelift_codegen::ir::immediates::Ieee64; use cranelift_codegen::ir::{ - Block, BlockArg, InstBuilder, MemFlags, StackSlot, StackSlotData, StackSlotKind, types, + AbiParam, Block, BlockArg, InstBuilder, MemFlags, Signature, StackSlot, StackSlotData, + StackSlotKind, types, }; -use cranelift_codegen::isa::OwnedTargetIsa; +use cranelift_codegen::isa::{CallConv, OwnedTargetIsa}; use cranelift_codegen::settings::{self, Configurable}; use cranelift_frontend::{FunctionBuilder, FunctionBuilderContext}; use cranelift_jit::{JITBuilder, JITModule}; @@ -49,11 +54,23 @@ use std::sync::atomic::{AtomicU64, Ordering}; static CRANELIFT_TRACE_ID: AtomicU64 = AtomicU64::new(1); static CRANELIFT_JIT_ISA: OnceLock> = OnceLock::new(); +static CRANELIFT_TAIL_ISA: OnceLock> = OnceLock::new(); + +const MAX_INHERITED_ENTRY_VALUES: usize = 256; +const INHERITED_STATE_ACTIVE_OFFSET: i32 = 0; +const INHERITED_STATE_FRAME_KEY_OFFSET: i32 = 8; +const INHERITED_STATE_STACK_BASE_OFFSET: i32 = 16; +const INHERITED_STATE_LOCAL_BASE_OFFSET: i32 = 24; +const INHERITED_STATE_DYNAMIC_TARGET_OFFSET: i32 = 32; +const INHERITED_STATE_TARGET_IP_OFFSET: i32 = 40; +const INHERITED_STATE_VALUE_COUNT_OFFSET: i32 = 48; +const INHERITED_STATE_VALUES_OFFSET: i32 = 56; type TaggedConstants = (Box<[Value]>, HashMap); pub(crate) struct CompiledTrace { pub(crate) entry: *const u8, + pub(crate) tail_entry: *const u8, pub(crate) keepalive: TraceKeepAlive, pub(crate) code: Vec, pub(crate) lowering_kind: TraceLoweringKind, @@ -62,6 +79,7 @@ pub(crate) struct CompiledTrace { pub(crate) struct TraceKeepAlive { exec: ExecutableBuffer, _tagged_constants: Box<[Value]>, + dependencies: Vec, } impl TraceKeepAlive { @@ -69,6 +87,7 @@ impl TraceKeepAlive { Ok(Self { exec: ExecutableBuffer::new(code)?, _tagged_constants: tagged_constants, + dependencies: Vec::new(), }) } @@ -77,11 +96,470 @@ impl TraceKeepAlive { } } +pub(crate) struct CompiledTailFunction { + entry: *const u8, + _keepalive: TraceKeepAlive, + code: Vec, +} + +impl CompiledTailFunction { + pub(crate) fn entry(&self) -> *const u8 { + self.entry + } + + pub(crate) fn code_len(&self) -> usize { + self.code.len() + } + + pub(crate) fn into_parts(self) -> (*const u8, TraceKeepAlive, Vec) { + (self.entry, self._keepalive, self.code) + } +} + +fn tail_entry_signature(pointer_type: cranelift_codegen::ir::Type) -> Signature { + let mut signature = Signature::new(CallConv::Tail); + signature.params.push(AbiParam::new(pointer_type)); + signature.returns.push(AbiParam::new(types::I32)); + signature +} + +fn inherited_tail_entry_signature(pointer_type: cranelift_codegen::ir::Type) -> Signature { + let mut signature = tail_entry_signature(pointer_type); + signature.params.push(AbiParam::new(pointer_type)); + signature +} + +fn compile_standalone_native_function( + prefix: &str, + signature: impl FnOnce(cranelift_codegen::ir::Type, CallConv) -> Signature, + lower: impl FnOnce(&mut FunctionBuilder<'_>, cranelift_codegen::ir::Type, CallConv) -> VmResult<()>, +) -> VmResult { + let isa = native_tail_isa()?; + let jit_builder = JITBuilder::with_isa(isa, cranelift_module::default_libcall_names()); + let mut module = JITModule::new(jit_builder); + let pointer_type = module.target_config().pointer_type(); + let default_call_conv = module.target_config().default_call_conv; + let mut ctx = module.make_context(); + ctx.func.signature = signature(pointer_type, default_call_conv); + let function_id = CRANELIFT_TRACE_ID.fetch_add(1, Ordering::Relaxed); + let function_name = format!("{prefix}_{function_id}"); + let func_id = module + .declare_function(&function_name, Linkage::Local, &ctx.func.signature) + .map_err(|err| VmError::JitNative(format!("declare {prefix} failed: {err}")))?; + { + let mut fb_ctx = FunctionBuilderContext::new(); + let mut builder = FunctionBuilder::new(&mut ctx.func, &mut fb_ctx); + lower(&mut builder, pointer_type, default_call_conv)?; + builder.seal_all_blocks(); + builder.finalize(); + } + module + .define_function(func_id, &mut ctx) + .map_err(|err| VmError::JitNative(format!("define {prefix} failed: {err}")))?; + let code_len = ctx + .compiled_code() + .ok_or_else(|| VmError::JitNative(format!("{prefix} produced no machine code")))? + .code_buffer() + .len(); + module.clear_context(&mut ctx); + module + .finalize_definitions() + .map_err(|err| VmError::JitNative(format!("finalize {prefix} failed: {err}")))?; + let module_entry = module.get_finalized_function(func_id); + let code = unsafe { std::slice::from_raw_parts(module_entry, code_len).to_vec() }; + let keepalive = TraceKeepAlive::from_code(&code, Box::new([]))?; + let entry = keepalive.entry(); + Ok(CompiledTailFunction { + entry, + _keepalive: keepalive, + code, + }) +} + +pub(crate) fn compile_tail_status_body(status: i32) -> VmResult { + compile_standalone_native_function( + "pd_vm_tail_status", + |pointer_type, _| tail_entry_signature(pointer_type), + move |builder, _, _| { + let entry = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.switch_to_block(entry); + let status = builder.ins().iconst(types::I32, i64::from(status)); + builder.ins().return_(&[status]); + Ok(()) + }, + ) +} + +pub(crate) fn compile_tail_side_link_body( + slot_address: usize, + deopt_status: i32, +) -> VmResult { + compile_standalone_native_function( + "pd_vm_tail_side_link", + |pointer_type, _| tail_entry_signature(pointer_type), + move |builder, pointer_type, _| { + let entry = builder.create_block(); + let deopt = builder.create_block(); + let linked = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.switch_to_block(entry); + let vm_ptr = builder.block_params(entry)[0]; + let slot_address = iconst_ptr_from_addr(builder, pointer_type, slot_address)?; + let target = builder + .ins() + .atomic_load(pointer_type, MemFlags::new(), slot_address); + let is_null = builder.ins().icmp_imm(IntCC::Equal, target, 0); + builder.ins().brif(is_null, deopt, &[], linked, &[]); + + builder.switch_to_block(deopt); + let status = builder.ins().iconst(types::I32, i64::from(deopt_status)); + builder.ins().return_(&[status]); + + builder.switch_to_block(linked); + let signature = builder.import_signature(tail_entry_signature(pointer_type)); + builder + .ins() + .return_call_indirect(signature, target, &[vm_ptr]); + Ok(()) + }, + ) +} + +pub(crate) fn compile_system_tail_wrapper(root_entry: *const u8) -> VmResult { + let root_entry = root_entry as usize; + compile_standalone_native_function( + "pd_vm_tail_wrapper", + entry_signature, + move |builder, pointer_type, _| { + let entry = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.switch_to_block(entry); + let vm_ptr = builder.block_params(entry)[0]; + let root_entry = iconst_ptr_from_addr(builder, pointer_type, root_entry)?; + let signature = builder.import_signature(tail_entry_signature(pointer_type)); + let call = builder + .ins() + .call_indirect(signature, root_entry, &[vm_ptr]); + let status = builder.inst_results(call)[0]; + builder.ins().return_(&[status]); + Ok(()) + }, + ) +} + +pub(crate) fn compile_system_inherited_tail_wrapper( + root_entry: *const u8, +) -> VmResult { + let root_entry = root_entry as usize; + compile_standalone_native_function( + "pd_vm_inherited_tail_wrapper", + entry_signature, + move |builder, pointer_type, _| { + let entry = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.switch_to_block(entry); + let vm_ptr = builder.block_params(entry)[0]; + let pointer_bytes = pointer_type.bits() / 8; + let packet_bytes = pointer_bytes + .checked_mul((MAX_INHERITED_ENTRY_VALUES + 7) as u32) + .ok_or_else(|| { + VmError::JitNative("native inherited-state packet is too large".to_string()) + })?; + let packet = builder.create_sized_stack_slot(StackSlotData::new( + StackSlotKind::ExplicitSlot, + packet_bytes, + pointer_bytes.trailing_zeros() as u8, + )); + let packet_ptr = builder.ins().stack_addr(pointer_type, packet, 0); + let inactive = builder.ins().iconst(pointer_type, 0); + builder.ins().store( + MemFlags::new(), + inactive, + packet_ptr, + INHERITED_STATE_ACTIVE_OFFSET, + ); + builder.ins().store( + MemFlags::new(), + inactive, + packet_ptr, + INHERITED_STATE_DYNAMIC_TARGET_OFFSET, + ); + let root_entry = iconst_ptr_from_addr(builder, pointer_type, root_entry)?; + let signature = builder.import_signature(inherited_tail_entry_signature(pointer_type)); + let call = builder + .ins() + .call_indirect(signature, root_entry, &[vm_ptr, packet_ptr]); + let status = builder.inst_results(call)[0]; + builder.ins().return_(&[status]); + Ok(()) + }, + ) +} + +pub(crate) fn compile_tail_trace_dispatcher( + trace_entry: *const u8, + trace_id: usize, + links: &[(i32, usize)], +) -> VmResult { + let trace_entry = trace_entry as usize; + let links = links.to_vec(); + let direct_link_offset = detect_native_stack_layout()?.vm_jit_native_direct_link_count_offset; + let active_trace_offset = + detect_native_stack_layout()?.vm_jit_native_active_direct_trace_id_offset; + compile_standalone_native_function( + "pd_vm_tail_trace_dispatch", + |pointer_type, _| inherited_tail_entry_signature(pointer_type), + move |builder, pointer_type, _default_call_conv| { + let entry = builder.create_block(); + let return_status = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.append_block_param(return_status, types::I32); + builder.switch_to_block(entry); + let vm_ptr = builder.block_params(entry)[0]; + let inherited_state_ptr = builder.block_params(entry)[1]; + let trace_id = i64::try_from(trace_id) + .map_err(|_| VmError::JitNative("native trace id exceeds i64".to_string()))?; + let trace_id = builder.ins().iconst(pointer_type, trace_id); + builder + .ins() + .store(MemFlags::new(), trace_id, vm_ptr, active_trace_offset); + let trace_entry = iconst_ptr_from_addr(builder, pointer_type, trace_entry)?; + let trace_signature = + builder.import_signature(inherited_tail_entry_signature(pointer_type)); + let call = builder.ins().call_indirect( + trace_signature, + trace_entry, + &[vm_ptr, inherited_state_ptr], + ); + let status = builder.inst_results(call)[0]; + + let static_dispatch = builder.create_block(); + let dynamic_check = builder.create_block(); + let dynamic_linked = builder.create_block(); + builder.append_block_param(static_dispatch, types::I32); + let is_linked_continue = builder.ins().icmp_imm( + IntCC::Equal, + status, + i64::from(crate::vm::native::STATUS_LINKED_CONTINUE), + ); + builder.ins().brif( + is_linked_continue, + dynamic_check, + &[], + static_dispatch, + &[status.into()], + ); + + builder.switch_to_block(dynamic_check); + let dynamic_target = builder.ins().load( + pointer_type, + MemFlags::new(), + inherited_state_ptr, + INHERITED_STATE_DYNAMIC_TARGET_OFFSET, + ); + let dynamic_target_is_null = builder.ins().icmp_imm(IntCC::Equal, dynamic_target, 0); + builder.ins().brif( + dynamic_target_is_null, + return_status, + &[status.into()], + dynamic_linked, + &[], + ); + + builder.switch_to_block(dynamic_linked); + let direct_count = + builder + .ins() + .load(types::I64, MemFlags::new(), vm_ptr, direct_link_offset); + let direct_count = builder.ins().iadd_imm(direct_count, 1); + builder + .ins() + .store(MemFlags::new(), direct_count, vm_ptr, direct_link_offset); + let dynamic_tail_signature = + builder.import_signature(inherited_tail_entry_signature(pointer_type)); + builder.ins().return_call_indirect( + dynamic_tail_signature, + dynamic_target, + &[vm_ptr, inherited_state_ptr], + ); + + builder.switch_to_block(static_dispatch); + let status = builder.block_params(static_dispatch)[0]; + for (linked_status, slot_address) in links { + let slot_check = builder.create_block(); + let next = builder.create_block(); + let linked = builder.create_block(); + builder.append_block_param(next, types::I32); + let matches = + builder + .ins() + .icmp_imm(IntCC::Equal, status, i64::from(linked_status)); + builder + .ins() + .brif(matches, slot_check, &[], next, &[status.into()]); + + builder.switch_to_block(slot_check); + let slot_address = iconst_ptr_from_addr(builder, pointer_type, slot_address)?; + let target = builder + .ins() + .atomic_load(pointer_type, MemFlags::new(), slot_address); + let is_null = builder.ins().icmp_imm(IntCC::Equal, target, 0); + builder + .ins() + .brif(is_null, return_status, &[status.into()], linked, &[]); + + builder.switch_to_block(linked); + let direct_count = + builder + .ins() + .load(types::I64, MemFlags::new(), vm_ptr, direct_link_offset); + let direct_count = builder.ins().iadd_imm(direct_count, 1); + builder + .ins() + .store(MemFlags::new(), direct_count, vm_ptr, direct_link_offset); + let tail_signature = + builder.import_signature(inherited_tail_entry_signature(pointer_type)); + builder.ins().return_call_indirect( + tail_signature, + target, + &[vm_ptr, inherited_state_ptr], + ); + + builder.switch_to_block(next); + } + + builder.ins().jump(return_status, &[status.into()]); + builder.switch_to_block(return_status); + let status = builder.block_params(return_status)[0]; + builder.ins().return_(&[status]); + Ok(()) + }, + ) +} + +fn tail_owned_entry_signature(pointer_type: cranelift_codegen::ir::Type) -> Signature { + let mut signature = tail_entry_signature(pointer_type); + signature.params.push(AbiParam::new(pointer_type)); + signature +} + +fn system_owned_entry_signature( + pointer_type: cranelift_codegen::ir::Type, + call_conv: CallConv, +) -> Signature { + let mut signature = entry_signature(pointer_type, call_conv); + signature.params.push(AbiParam::new(pointer_type)); + signature +} + +pub(crate) fn compile_tail_owned_side_link_body( + slot_address: usize, + deopt_status: i32, +) -> VmResult { + compile_standalone_native_function( + "pd_vm_tail_owned_side_link", + |pointer_type, _| tail_owned_entry_signature(pointer_type), + move |builder, pointer_type, _| { + let entry = builder.create_block(); + let deopt = builder.create_block(); + let linked = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.switch_to_block(entry); + let vm_ptr = builder.block_params(entry)[0]; + let owned_slot = builder.block_params(entry)[1]; + let slot_address = iconst_ptr_from_addr(builder, pointer_type, slot_address)?; + let target = builder + .ins() + .atomic_load(pointer_type, MemFlags::new(), slot_address); + let is_null = builder.ins().icmp_imm(IntCC::Equal, target, 0); + builder.ins().brif(is_null, deopt, &[], linked, &[]); + + builder.switch_to_block(deopt); + let status = builder.ins().iconst(types::I32, i64::from(deopt_status)); + builder.ins().return_(&[status]); + + builder.switch_to_block(linked); + let signature = builder.import_signature(tail_owned_entry_signature(pointer_type)); + builder + .ins() + .return_call_indirect(signature, target, &[vm_ptr, owned_slot]); + Ok(()) + }, + ) +} + +pub(crate) fn compile_tail_owned_clear_body(success_status: i32) -> VmResult { + compile_standalone_native_function( + "pd_vm_tail_owned_clear", + |pointer_type, _| tail_owned_entry_signature(pointer_type), + move |builder, pointer_type, call_conv| { + let entry = builder.create_block(); + let success = builder.create_block(); + let failure = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.append_block_param(failure, types::I32); + builder.switch_to_block(entry); + let owned_slot = builder.block_params(entry)[1]; + let helper = + iconst_ptr_from_addr(builder, pointer_type, clear_value_slot_entry_address())?; + let helper_signature = + builder.import_signature(value_slot_signature(pointer_type, call_conv)); + let call = builder + .ins() + .call_indirect(helper_signature, helper, &[owned_slot]); + let helper_status = builder.inst_results(call)[0]; + let succeeded = + builder + .ins() + .icmp_imm(IntCC::Equal, helper_status, i64::from(STATUS_CONTINUE)); + builder + .ins() + .brif(succeeded, success, &[], failure, &[helper_status.into()]); + + builder.switch_to_block(success); + let status = builder.ins().iconst(types::I32, i64::from(success_status)); + builder.ins().return_(&[status]); + + builder.switch_to_block(failure); + let status = builder.block_params(failure)[0]; + builder.ins().return_(&[status]); + Ok(()) + }, + ) +} + +pub(crate) fn compile_system_owned_tail_wrapper( + root_entry: *const u8, +) -> VmResult { + let root_entry = root_entry as usize; + compile_standalone_native_function( + "pd_vm_tail_owned_wrapper", + system_owned_entry_signature, + move |builder, pointer_type, _| { + let entry = builder.create_block(); + builder.append_block_params_for_function_params(entry); + builder.switch_to_block(entry); + let vm_ptr = builder.block_params(entry)[0]; + let owned_slot = builder.block_params(entry)[1]; + let root_entry = iconst_ptr_from_addr(builder, pointer_type, root_entry)?; + let signature = builder.import_signature(tail_owned_entry_signature(pointer_type)); + let call = builder + .ins() + .call_indirect(signature, root_entry, &[vm_ptr, owned_slot]); + let status = builder.inst_results(call)[0]; + builder.ins().return_(&[status]); + Ok(()) + }, + ) +} + fn try_compile_ssa_trace( trace: &JitTrace, ssa: &SsaTrace, + internal_links: &[FusedRegionLink], interrupt_settings: Option, - profile: NativeCompileProfile, + _profile: NativeCompileProfile, drop_contract_events_enabled: bool, ) -> VmResult> { if drop_contract_events_enabled { @@ -93,7 +571,7 @@ fn try_compile_ssa_trace( let layout = detect_native_stack_layout()?; let offsets = resolve_offsets(layout)?; - let isa = native_isa(profile)?; + let isa = native_tail_isa()?; let jit_builder = JITBuilder::with_isa(isa, cranelift_module::default_libcall_names()); let mut module = JITModule::new(jit_builder); @@ -101,9 +579,13 @@ fn try_compile_ssa_trace( let call_conv = module.target_config().default_call_conv; let mut ctx = module.make_context(); - ctx.func.signature = entry_signature(pointer_type, call_conv); + ctx.func.signature = inherited_tail_entry_signature(pointer_type); let clone_value_sig = clone_value_signature(pointer_type, call_conv); let non_yielding_host_call_sig = non_yielding_host_call_signature(pointer_type, call_conv); + let non_yielding_scalar_host_call_sig = + non_yielding_scalar_host_call_signature(pointer_type, call_conv); + let non_yielding_i64_host_call_sig = + non_yielding_i64_host_call_signature(pointer_type, call_conv); let value_slot_sig = value_slot_signature(pointer_type, call_conv); let value_eq_sig = value_eq_signature(pointer_type, call_conv); let value_len_sig = value_len_signature(pointer_type, call_conv); @@ -121,8 +603,8 @@ fn try_compile_ssa_trace( let map_set_sig = map_set_signature(pointer_type, call_conv); let sparse_restore_exit_sig = sparse_restore_exit_signature(pointer_type, call_conv); let frame_state_sig = frame_state_signature(pointer_type, call_conv); - let leave_frame_sig = leave_frame_signature(pointer_type, call_conv); - let enter_call_value_sig = enter_call_value_signature(pointer_type, call_conv); + let leave_frame_sig = leave_frame_inherited_signature(pointer_type, call_conv); + let enter_call_value_sig = enter_call_value_inherited_signature(pointer_type, call_conv); let resume_linked_trace_sig = entry_signature(pointer_type, call_conv); let string_contains_sig = string_contains_signature(pointer_type, call_conv); @@ -187,6 +669,9 @@ fn try_compile_ssa_trace( value_eq_ref: b.import_signature(value_eq_sig), value_len_ref: b.import_signature(value_len_sig), non_yielding_host_call_ref: b.import_signature(non_yielding_host_call_sig), + non_yielding_scalar_host_call_ref: b + .import_signature(non_yielding_scalar_host_call_sig), + non_yielding_i64_host_call_ref: b.import_signature(non_yielding_i64_host_call_sig), clear_value_slot_ref: b.import_signature(value_slot_sig), box_heap_value_ref: b.import_signature(box_heap_value_sig), map_has_ref: b.import_signature(map_has_sig), @@ -208,6 +693,8 @@ fn try_compile_ssa_trace( value_eq: value_eq_entry_address(), value_len: value_len_entry_address(), non_yielding_host_call: non_yielding_host_call_entry_address(), + non_yielding_scalar_host_call: non_yielding_scalar_host_call_entry_address(), + non_yielding_i64_host_call: non_yielding_i64_host_call_entry_address(), clear_value_slot: clear_value_slot_entry_address(), box_heap_value: write_heap_value_to_slot_entry_address(), map_has: map_has_entry_address(), @@ -219,8 +706,8 @@ fn try_compile_ssa_trace( array_set: array_set_entry_address(), map_set: map_set_entry_address(), sparse_restore_exit: restore_active_sparse_exit_state_entry_address(), - leave_frame: leave_frame_entry_address(), - enter_call_value: enter_call_value_entry_address(), + leave_frame: leave_frame_inherited_entry_address(), + enter_call_value: enter_call_value_inherited_entry_address(), resume_linked_trace: resume_linked_trace_entry_address(), }; @@ -240,8 +727,17 @@ fn try_compile_ssa_trace( } } let borrowed_array_gets = borrowed_array_get_outputs(ssa); - let owned_value_temps = - allocate_owned_value_temps(&mut b, ssa, layout.value.size, &borrowed_array_gets)?; + let owned_value_temps = allocate_owned_value_temps( + &mut b, + ssa, + layout.value.size, + &borrowed_array_gets, + &value_reprs, + )?; + let internal_link_slots = internal_links + .iter() + .map(|link| allocate_internal_link_slots(&mut b, link, layout.value.size)) + .collect::>>()?; let mut block_handles = HashMap::new(); for block in &ssa.blocks { @@ -276,6 +772,9 @@ fn try_compile_ssa_trace( let call_value_block = call_value_exits .contains_key(&exit.id) .then(|| b.create_block()); + let interrupt_block = (interrupt_settings.is_some() + && internal_links.iter().any(|link| link.exit == exit.id)) + .then(|| b.create_block()); for value in &inputs { let Some(repr) = value_reprs.get(value).copied() else { return Ok(None); @@ -288,6 +787,9 @@ fn try_compile_ssa_trace( if let Some(call_value_block) = call_value_block { b.append_block_param(call_value_block, ty); } + if let Some(interrupt_block) = interrupt_block { + b.append_block_param(interrupt_block, ty); + } } exit_specs.insert( exit.id, @@ -295,6 +797,7 @@ fn try_compile_ssa_trace( trace_exit_block, halted_block, call_value_block, + interrupt_block, inputs, }, ); @@ -303,15 +806,129 @@ fn try_compile_ssa_trace( b.switch_to_block(entry_block); b.append_block_params_for_function_params(entry_block); let vm_ptr = b.block_params(entry_block)[0]; - let entry_local_count = ssa + let inherited_state_ptr = b.block_params(entry_block)[1]; + let entry_ssa_block = ssa .blocks .get(ssa.entry.index()) - .and_then(|block| block.params.len().checked_sub(ssa.entry_stack_depth)) + .ok_or_else(|| VmError::JitNative("SSA entry block is missing".to_string()))?; + let entry_local_count = entry_ssa_block + .params + .len() + .checked_sub(ssa.entry_stack_depth) .ok_or_else(|| { VmError::JitNative( "SSA entry stack depth exceeds entry parameter count".to_string(), ) })?; + let entry_ready = b.create_block(); + b.append_block_param(entry_ready, pointer_type); + b.append_block_param(entry_ready, pointer_type); + for param in &entry_ssa_block.params { + let ty = ssa_type(pointer_type, param.value.repr).ok_or_else(|| { + VmError::JitNative("SSA inherited entry parameter type is unsupported".to_string()) + })?; + b.append_block_param(entry_ready, ty); + } + + let normal_entry = b.create_block(); + if entry_ssa_block.params.len() <= MAX_INHERITED_ENTRY_VALUES { + let inherited_candidate = b.create_block(); + let inherited_entry = b.create_block(); + let active = b.ins().load( + pointer_type, + MemFlags::new(), + inherited_state_ptr, + INHERITED_STATE_ACTIVE_OFFSET, + ); + let has_inherited_state = b.ins().icmp_imm(IntCC::NotEqual, active, 0); + b.ins().brif( + has_inherited_state, + inherited_candidate, + &[], + normal_entry, + &[], + ); + + b.switch_to_block(inherited_candidate); + let inherited_frame_key = b.ins().load( + pointer_type, + MemFlags::new(), + inherited_state_ptr, + INHERITED_STATE_FRAME_KEY_OFFSET, + ); + let expected_frame_key = b.ins().iconst(pointer_type, trace.frame_key as i64); + let frame_matches = b + .ins() + .icmp(IntCC::Equal, inherited_frame_key, expected_frame_key); + let inherited_target_ip = b.ins().load( + pointer_type, + MemFlags::new(), + inherited_state_ptr, + INHERITED_STATE_TARGET_IP_OFFSET, + ); + let expected_target_ip = b.ins().iconst( + pointer_type, + i64::try_from(trace.root_ip).map_err(|_| { + VmError::JitNative("SSA inherited target ip exceeds i64".to_string()) + })?, + ); + let target_matches = + b.ins() + .icmp(IntCC::Equal, inherited_target_ip, expected_target_ip); + let inherited_value_count = b.ins().load( + pointer_type, + MemFlags::new(), + inherited_state_ptr, + INHERITED_STATE_VALUE_COUNT_OFFSET, + ); + let expected_value_count = b.ins().iconst( + pointer_type, + i64::try_from(entry_ssa_block.params.len()).map_err(|_| { + VmError::JitNative("SSA inherited value count exceeds i64".to_string()) + })?, + ); + let value_count_matches = + b.ins() + .icmp(IntCC::Equal, inherited_value_count, expected_value_count); + let metadata_matches = b.ins().band(frame_matches, target_matches); + let metadata_matches = b.ins().band(metadata_matches, value_count_matches); + b.ins() + .brif(metadata_matches, inherited_entry, &[], normal_entry, &[]); + + b.switch_to_block(inherited_entry); + let active_stack_base = b.ins().load( + pointer_type, + MemFlags::new(), + inherited_state_ptr, + INHERITED_STATE_STACK_BASE_OFFSET, + ); + let active_local_base = b.ins().load( + pointer_type, + MemFlags::new(), + inherited_state_ptr, + INHERITED_STATE_LOCAL_BASE_OFFSET, + ); + let inherited_args = build_inherited_entry_args( + &mut b, + inherited_state_ptr, + pointer_type, + entry_ssa_block.params.len(), + )?; + let inactive = b.ins().iconst(pointer_type, 0); + b.ins().store( + MemFlags::new(), + inactive, + inherited_state_ptr, + INHERITED_STATE_ACTIVE_OFFSET, + ); + let mut ready_args = vec![active_stack_base.into(), active_local_base.into()]; + ready_args.extend(ssa_block_args(inherited_args)); + b.ins().jump(entry_ready, &ready_args); + } else { + b.ins().jump(normal_entry, &[]); + } + + b.switch_to_block(normal_entry); let (active_stack_base, active_local_base) = emit_frame_state_entry_guard( &mut b, vm_ptr, @@ -324,8 +941,31 @@ fn try_compile_ssa_trace( ssa.entry_stack_depth, entry_local_count, )?; + let normal_args = build_entry_args( + &mut b, + vm_ptr, + pointer_type, + layout, + offsets, + active_stack_base, + active_local_base, + ssa.entry_stack_depth, + entry_local_count, + )?; + let mut ready_args = vec![active_stack_base.into(), active_local_base.into()]; + ready_args.extend(ssa_block_args(normal_args)); + b.ins().jump(entry_ready, &ready_args); + + b.switch_to_block(entry_ready); + let ready_params = b.block_params(entry_ready).to_vec(); + let active_stack_base = ready_params[0]; + let active_local_base = ready_params[1]; + let interrupt_ops_to_advance = u32::try_from(trace.op_names.len().max(1)) + .map_err(|_| VmError::JitNative("trace op count exceeds u32".to_string()))?; let lower_ctx = SsaLowerCtx { vm_ptr, + inherited_state_ptr, + frame_key: trace.frame_key, active_stack_base, active_local_base, exit_block, @@ -343,6 +983,8 @@ fn try_compile_ssa_trace( borrowed_array_gets: &borrowed_array_gets, value_reprs: &value_reprs, tagged_constant_addrs: &tagged_constant_addrs, + interrupt_settings, + interrupt_ops_to_advance, }; let root_ip = b.ins().iconst( pointer_type, @@ -354,24 +996,11 @@ fn try_compile_ssa_trace( let entry_handle = *block_handles .get(&ssa.entry) .ok_or_else(|| VmError::JitNative("SSA entry block handle missing".to_string()))?; - let entry_args = build_entry_args( - &mut b, - vm_ptr, - pointer_type, - layout, - offsets, - active_stack_base, - active_local_base, - ssa.entry_stack_depth, - entry_local_count, - )?; init_owned_value_temps(&mut b, pointer_type, layout.value, &owned_value_temps)?; - let entry_args = ssa_block_args(entry_args); + let entry_args = ssa_block_args(ready_params[2..].to_vec()); b.ins().jump(entry_handle, &entry_args); let charge_blocks = ssa_interrupt_charge_blocks(ssa); - let ops_to_advance = u32::try_from(trace.op_names.len().max(1)) - .map_err(|_| VmError::JitNative("trace op count exceeds u32".to_string()))?; for block in &ssa.blocks { let handle = *block_handles .get(&block.id) @@ -382,7 +1011,8 @@ fn try_compile_ssa_trace( for (param, lowered) in block.params.iter().zip(block_params.iter().copied()) { values.insert(param.value.id, lowered); } - if charge_blocks.contains(&block.id) + if internal_links.is_empty() + && charge_blocks.contains(&block.id) && let Some(settings) = interrupt_settings { emit_interrupt_tick_inline( @@ -390,7 +1020,7 @@ fn try_compile_ssa_trace( vm_ptr, exit_block, offsets, - ops_to_advance, + lower_ctx.interrupt_ops_to_advance, settings, ); } @@ -406,6 +1036,10 @@ fn try_compile_ssa_trace( &values, &block_handles, &exit_specs, + InternalRegionLinks { + links: internal_links, + slots: &internal_link_slots, + }, )?; } @@ -436,6 +1070,9 @@ fn try_compile_ssa_trace( }, )?; } + if spec.interrupt_block.is_some() { + lower_ssa_exit_block(&mut b, lower_ctx, exit, spec, SsaExitAction::InterruptYield)?; + } } b.switch_to_block(exit_block); @@ -479,6 +1116,7 @@ fn try_compile_ssa_trace( Ok(Some(CompiledTrace { entry, + tail_entry: entry, keepalive, code, lowering_kind: TraceLoweringKind::Ssa, @@ -490,6 +1128,7 @@ struct SsaExitLowering { trace_exit_block: Block, halted_block: Block, call_value_block: Option, + interrupt_block: Option, inputs: Vec, } @@ -504,6 +1143,7 @@ enum SsaExitAction { call_ip: usize, resume_ip: usize, }, + InterruptYield, } #[derive(Clone, Copy)] @@ -512,6 +1152,8 @@ struct SsaDeoptHelperRefs { value_eq_ref: cranelift_codegen::ir::SigRef, value_len_ref: cranelift_codegen::ir::SigRef, non_yielding_host_call_ref: cranelift_codegen::ir::SigRef, + non_yielding_scalar_host_call_ref: cranelift_codegen::ir::SigRef, + non_yielding_i64_host_call_ref: cranelift_codegen::ir::SigRef, clear_value_slot_ref: cranelift_codegen::ir::SigRef, box_heap_value_ref: cranelift_codegen::ir::SigRef, map_has_ref: cranelift_codegen::ir::SigRef, @@ -535,6 +1177,8 @@ struct SsaDeoptHelperAddrs { value_eq: usize, value_len: usize, non_yielding_host_call: usize, + non_yielding_scalar_host_call: usize, + non_yielding_i64_host_call: usize, clear_value_slot: usize, box_heap_value: usize, map_has: usize, @@ -580,6 +1224,7 @@ struct SsaStringHelperAddrs { enum SsaTempValueSlotKey { Output(SsaValueId), HostArgs(SsaValueId), + HostScalarResult(SsaValueId), MapKeyBox(SsaValueId), MutationArgBox(SsaValueId, u8), } @@ -590,9 +1235,27 @@ struct SsaOwnedValueTemps { slots: HashMap, } +#[derive(Clone, Copy)] +struct InternalRegionLinks<'a> { + links: &'a [FusedRegionLink], + slots: &'a [Vec>], +} + +impl<'a> InternalRegionLinks<'a> { + fn find(self, exit: SsaExitId) -> Option<(&'a FusedRegionLink, &'a [Option])> { + self.links + .iter() + .zip(self.slots) + .find(|(link, _)| link.exit == exit) + .map(|(link, slots)| (link, slots.as_slice())) + } +} + #[derive(Clone, Copy)] struct SsaLowerCtx<'a> { vm_ptr: cranelift_codegen::ir::Value, + inherited_state_ptr: cranelift_codegen::ir::Value, + frame_key: u64, active_stack_base: cranelift_codegen::ir::Value, active_local_base: cranelift_codegen::ir::Value, exit_block: Block, @@ -610,6 +1273,8 @@ struct SsaLowerCtx<'a> { borrowed_array_gets: &'a BTreeSet, value_reprs: &'a HashMap, tagged_constant_addrs: &'a HashMap, + interrupt_settings: Option, + interrupt_ops_to_advance: u32, } #[derive(Clone, Copy)] @@ -670,7 +1335,9 @@ fn ssa_trace_supported(ssa: &SsaTrace) -> bool { | SsaInstKind::StringConcat { .. } | SsaInstKind::BytesConcat { .. } | SsaInstKind::BytesFromArrayU8 { .. } + | SsaInstKind::BytesToUtf8Ascii { .. } | SsaInstKind::BytesToArrayU8 { .. } + | SsaInstKind::ArrayNew | SsaInstKind::ArrayLen { .. } | SsaInstKind::ArrayGet { .. } | SsaInstKind::ArrayHas { .. } @@ -785,11 +1452,52 @@ fn prepare_tagged_constants(ssa: &SsaTrace) -> VmResult { Ok((values, out)) } +fn scalar_host_return_type(repr: SsaValueRepr) -> VmResult { + match repr { + SsaValueRepr::I64 => Ok(ValueType::Int), + SsaValueRepr::F64 => Ok(ValueType::Float), + SsaValueRepr::Bool => Ok(ValueType::Bool), + _ => Err(VmError::JitNative( + "SSA scalar host-call result representation is unsupported".to_string(), + )), + } +} + +fn ssa_load_scalar_host_result( + b: &mut FunctionBuilder, + repr: SsaValueRepr, + out: cranelift_codegen::ir::Value, +) -> VmResult { + match repr { + SsaValueRepr::I64 => Ok(b.ins().load(types::I64, MemFlags::new(), out, 0)), + SsaValueRepr::F64 => Ok(b.ins().load(types::F64, MemFlags::new(), out, 0)), + SsaValueRepr::Bool => { + let bits = b.ins().load(types::I64, MemFlags::new(), out, 0); + let raw = b.ins().ireduce(types::I8, bits); + Ok(b.ins().icmp_imm(IntCC::NotEqual, raw, 0)) + } + _ => Err(VmError::JitNative( + "SSA scalar host-call result representation is unsupported".to_string(), + )), + } +} + +fn host_call_has_direct_i64_args( + args: &[SsaValueId], + value_reprs: &HashMap, +) -> bool { + args.len() <= 2 + && args + .iter() + .all(|arg| value_reprs.get(arg) == Some(&SsaValueRepr::I64)) +} + fn allocate_owned_value_temps( b: &mut FunctionBuilder, ssa: &SsaTrace, value_size: i32, borrowed_array_gets: &BTreeSet, + value_reprs: &HashMap, ) -> VmResult { let mut ordered = Vec::new(); let mut slots = HashMap::new(); @@ -799,7 +1507,10 @@ fn allocate_owned_value_temps( let Some(output) = inst.output else { continue; }; + let scalar_host_call = matches!(inst.kind, SsaInstKind::HostCall { .. }) + && output.repr != SsaValueRepr::Tagged; if ssa_inst_requires_owned_value_slot(&inst.kind) + && !scalar_host_call && !borrowed_array_gets.contains(&output.id) { let slot = ssa_create_value_stack_slot(b, value_size)?; @@ -807,6 +1518,20 @@ fn allocate_owned_value_temps( slots.insert(SsaTempValueSlotKey::Output(output.id), slot); } if let SsaInstKind::HostCall { args, .. } = &inst.kind { + if scalar_host_call { + let scalar_slot = b.create_sized_stack_slot(StackSlotData::new( + StackSlotKind::ExplicitSlot, + std::mem::size_of::() as u32, + std::mem::align_of::().trailing_zeros() as u8, + )); + slots.insert( + SsaTempValueSlotKey::HostScalarResult(output.id), + scalar_slot, + ); + } + if scalar_host_call && host_call_has_direct_i64_args(args, value_reprs) { + continue; + } let arg_bytes = usize::try_from(value_size) .ok() .and_then(|value_size| value_size.checked_mul(args.len().max(1))) @@ -955,7 +1680,9 @@ fn ssa_inst_requires_owned_value_slot(kind: &SsaInstKind) -> bool { | SsaInstKind::ToString { .. } | SsaInstKind::StringSplitLiteral { .. } | SsaInstKind::BytesFromArrayU8 { .. } + | SsaInstKind::BytesToUtf8Ascii { .. } | SsaInstKind::BytesToArrayU8 { .. } + | SsaInstKind::ArrayNew | SsaInstKind::StringConcat { .. } | SsaInstKind::BytesConcat { .. } | SsaInstKind::HostCall { .. } @@ -1106,6 +1833,40 @@ fn ssa_backedge_targets( targets } +fn build_inherited_entry_args( + b: &mut FunctionBuilder, + inherited_state_ptr: cranelift_codegen::ir::Value, + pointer_type: cranelift_codegen::ir::Type, + entry_value_count: usize, +) -> VmResult> { + if entry_value_count > MAX_INHERITED_ENTRY_VALUES { + return Err(VmError::JitNative( + "SSA inherited entry value count exceeds packet capacity".to_string(), + )); + } + (0..entry_value_count) + .map(|index| { + let byte_offset = index + .checked_mul(std::mem::size_of::()) + .and_then(|offset| { + usize::try_from(INHERITED_STATE_VALUES_OFFSET) + .ok() + .and_then(|base| base.checked_add(offset)) + }) + .and_then(|offset| i32::try_from(offset).ok()) + .ok_or_else(|| { + VmError::JitNative("SSA inherited entry packet offset exceeds i32".to_string()) + })?; + Ok(b.ins().load( + pointer_type, + MemFlags::new(), + inherited_state_ptr, + byte_offset, + )) + }) + .collect() +} + #[allow(clippy::too_many_arguments)] fn build_entry_args( b: &mut FunctionBuilder, @@ -2244,11 +3005,97 @@ fn lower_ssa_inst( heap_refs, heap_addrs.pack_bytes, out_ptr, - values_len, - values_len, + values_len, + values_len, + )?; + clear_owned_value_temp_slot(b, pointer_type, helper_refs, helper_addrs, out)?; + ssa_store_heap_ptr_in_value(b, layout.value, out, layout.value.bytes_tag, out_raw); + b.ins().jump(cont, &[]); + + b.switch_to_block(fail); + ssa_emit_trace_exit_status(b, vm_ptr, exit_block, pointer_type, offsets, inst.ip)?; + + b.switch_to_block(cont); + out + } + SsaInstKind::BytesToUtf8Ascii { bytes } => { + let bytes = values[bytes]; + let bytes_data = ssa_load_heap_data_ptr(b, layout.value, bytes); + let bytes_ptr = b.ins().load( + pointer_type, + MemFlags::new(), + bytes_data, + layout.stack_vec.ptr_offset, + ); + let bytes_len = b.ins().load( + pointer_type, + MemFlags::new(), + bytes_data, + layout.stack_vec.len_offset, + ); + let out = owned_value_temp_slot_addr( + b, + pointer_type, + owned_value_temps, + SsaTempValueSlotKey::Output(output.id), + )?; + let validate_loop = b.create_block(); + let validate_step = b.create_block(); + let finish = b.create_block(); + let fail = b.create_block(); + let cont = b.create_block(); + b.append_block_param(validate_loop, pointer_type); + + let zero = b.ins().iconst(pointer_type, 0); + b.ins().jump(validate_loop, &[BlockArg::Value(zero)]); + + b.switch_to_block(validate_loop); + let index = b.block_params(validate_loop)[0]; + let done = b + .ins() + .icmp(IntCC::UnsignedGreaterThanOrEqual, index, bytes_len); + b.ins().brif(done, finish, &[], validate_step, &[]); + + b.switch_to_block(validate_step); + let byte_addr = b.ins().iadd(bytes_ptr, index); + let byte = b.ins().load(types::I8, MemFlags::new(), byte_addr, 0); + let is_ascii = b.ins().icmp_imm(IntCC::UnsignedLessThan, byte, 128); + let validate_next = b.create_block(); + b.ins().brif(is_ascii, validate_next, &[], fail, &[]); + + b.switch_to_block(validate_next); + let next_index = b.ins().iadd_imm(index, 1); + b.ins().jump(validate_loop, &[BlockArg::Value(next_index)]); + + b.switch_to_block(finish); + let out_ptr = ssa_call_alloc_buffer( + b, + pointer_type, + heap_refs, + heap_addrs, + heap_addrs.alloc_byte_buffer, + bytes_len, + )?; + ssa_call_copy_bytes( + b, + pointer_type, + heap_refs, + heap_addrs, + out_ptr, + bytes_ptr, + bytes_len, + )?; + let out_raw = ssa_call_pack_shared( + b, + pointer_type, + heap_refs, + heap_addrs.pack_string, + out_ptr, + bytes_len, + bytes_len, )?; clear_owned_value_temp_slot(b, pointer_type, helper_refs, helper_addrs, out)?; - ssa_store_heap_ptr_in_value(b, layout.value, out, layout.value.bytes_tag, out_raw); + ssa_store_heap_ptr_in_value(b, layout.value, out, layout.value.string_tag, out_raw); b.ins().jump(cont, &[]); b.switch_to_block(fail); @@ -2344,6 +3191,35 @@ fn lower_ssa_inst( b.switch_to_block(cont); out } + SsaInstKind::ArrayNew => { + let out = owned_value_temp_slot_addr( + b, + pointer_type, + owned_value_temps, + SsaTempValueSlotKey::Output(output.id), + )?; + let zero = b.ins().iconst(pointer_type, 0); + let out_ptr = ssa_call_alloc_buffer( + b, + pointer_type, + heap_refs, + heap_addrs, + heap_addrs.alloc_value_buffer, + zero, + )?; + let out_raw = ssa_call_pack_shared( + b, + pointer_type, + heap_refs, + heap_addrs.pack_array, + out_ptr, + zero, + zero, + )?; + clear_owned_value_temp_slot(b, pointer_type, helper_refs, helper_addrs, out)?; + ssa_store_heap_ptr_in_value(b, layout.value, out, layout.value.array_tag, out_raw); + out + } SsaInstKind::ArrayLen { array } => { let array = values[array]; let vec_ptr = ssa_load_heap_data_ptr(b, layout.value, array); @@ -2779,43 +3655,16 @@ fn lower_ssa_inst( out } SsaInstKind::HostCall { import, args } => { - let out = owned_value_temp_slot_addr( - b, - pointer_type, - owned_value_temps, - SsaTempValueSlotKey::Output(output.id), - )?; - let arg_values = owned_value_temp_slot_addr( - b, - pointer_type, - owned_value_temps, - SsaTempValueSlotKey::HostArgs(output.id), - )?; - for (index, arg) in args.iter().copied().enumerate() { - let repr = *value_reprs.get(&arg).ok_or_else(|| { - VmError::JitNative("SSA host-call argument representation missing".to_string()) - })?; - let value = values[&arg]; - let index_value = b.ins().iconst( - pointer_type, - i64::try_from(index).map_err(|_| { - VmError::JitNative("SSA host-call argument index out of range".to_string()) - })?, - ); - let addr = - ssa_value_addr(b, pointer_type, arg_values, index_value, layout.value.size); - match repr { - SsaValueRepr::Tagged => ssa_copy_value_bytes(b, value, addr, layout.value.size), - SsaValueRepr::I64 => ssa_store_int_in_value(b, layout.value, addr, value), - SsaValueRepr::F64 => ssa_store_float_in_value(b, layout.value, addr, value), - SsaValueRepr::Bool => ssa_store_bool_in_value(b, layout.value, addr, value), - SsaValueRepr::HeapPtr(tag) => { - let tag = ssa_heap_tag(layout.value, tag)?; - ssa_store_heap_ptr_in_value(b, layout.value, addr, tag, value); - } - } - } - clear_owned_value_temp_slot(b, pointer_type, helper_refs, helper_addrs, out)?; + let scalar_result = (output.repr != SsaValueRepr::Tagged) + .then(|| { + owned_value_temp_slot_addr( + b, + pointer_type, + owned_value_temps, + SsaTempValueSlotKey::HostScalarResult(output.id), + ) + }) + .transpose()?; let import = b.ins().iconst(pointer_type, i64::from(*import)); let argc = b.ins().iconst( pointer_type, @@ -2823,15 +3672,94 @@ fn lower_ssa_inst( VmError::JitNative("SSA host-call argument count out of range".to_string()) })?, ); - ssa_call_status_helper( - b, - exit_block, - pointer_type, - helper_refs.non_yielding_host_call_ref, - helper_addrs.non_yielding_host_call, - &[vm_ptr, import, arg_values, argc, out], - )?; - out + + if let Some(out) = scalar_result + && host_call_has_direct_i64_args(args, value_reprs) + { + let zero = b.ins().iconst(types::I64, 0); + let arg0 = args.first().map_or(zero, |arg| values[arg]); + let arg1 = args.get(1).map_or(zero, |arg| values[arg]); + let return_type = b + .ins() + .iconst(types::I64, scalar_host_return_type(output.repr)? as i64); + ssa_call_status_helper( + b, + exit_block, + pointer_type, + helper_refs.non_yielding_i64_host_call_ref, + helper_addrs.non_yielding_i64_host_call, + &[vm_ptr, import, arg0, arg1, argc, return_type, out], + )?; + ssa_load_scalar_host_result(b, output.repr, out)? + } else { + let arg_values = owned_value_temp_slot_addr( + b, + pointer_type, + owned_value_temps, + SsaTempValueSlotKey::HostArgs(output.id), + )?; + for (index, arg) in args.iter().copied().enumerate() { + let repr = *value_reprs.get(&arg).ok_or_else(|| { + VmError::JitNative( + "SSA host-call argument representation missing".to_string(), + ) + })?; + let value = values[&arg]; + let index_value = b.ins().iconst( + pointer_type, + i64::try_from(index).map_err(|_| { + VmError::JitNative( + "SSA host-call argument index out of range".to_string(), + ) + })?, + ); + let addr = + ssa_value_addr(b, pointer_type, arg_values, index_value, layout.value.size); + match repr { + SsaValueRepr::Tagged => { + ssa_copy_value_bytes(b, value, addr, layout.value.size) + } + SsaValueRepr::I64 => ssa_store_int_in_value(b, layout.value, addr, value), + SsaValueRepr::F64 => ssa_store_float_in_value(b, layout.value, addr, value), + SsaValueRepr::Bool => ssa_store_bool_in_value(b, layout.value, addr, value), + SsaValueRepr::HeapPtr(tag) => { + let tag = ssa_heap_tag(layout.value, tag)?; + ssa_store_heap_ptr_in_value(b, layout.value, addr, tag, value); + } + } + } + if let Some(out) = scalar_result { + let return_type = b + .ins() + .iconst(pointer_type, scalar_host_return_type(output.repr)? as i64); + ssa_call_status_helper( + b, + exit_block, + pointer_type, + helper_refs.non_yielding_scalar_host_call_ref, + helper_addrs.non_yielding_scalar_host_call, + &[vm_ptr, import, arg_values, argc, return_type, out], + )?; + ssa_load_scalar_host_result(b, output.repr, out)? + } else { + let out = owned_value_temp_slot_addr( + b, + pointer_type, + owned_value_temps, + SsaTempValueSlotKey::Output(output.id), + )?; + clear_owned_value_temp_slot(b, pointer_type, helper_refs, helper_addrs, out)?; + ssa_call_status_helper( + b, + exit_block, + pointer_type, + helper_refs.non_yielding_host_call_ref, + helper_addrs.non_yielding_host_call, + &[vm_ptr, import, arg_values, argc, out], + )?; + out + } + } } SsaInstKind::IntNeg { input } => b.ins().ineg(values[input]), SsaInstKind::IntAdd { lhs, rhs } => b.ins().iadd(values[lhs], values[rhs]), @@ -3059,13 +3987,104 @@ fn lower_ssa_inst( Ok(()) } +fn allocate_internal_link_slots( + b: &mut FunctionBuilder, + link: &FusedRegionLink, + value_size: i32, +) -> VmResult>> { + link.args + .iter() + .map(|materialization| match materialization { + SsaMaterialization::Value(_) => Ok(None), + SsaMaterialization::BoxInt(_) + | SsaMaterialization::BoxFloat(_) + | SsaMaterialization::BoxBool(_) => { + ssa_create_value_stack_slot(b, value_size).map(Some) + } + SsaMaterialization::BoxHeapPtr { .. } => Err(VmError::JitNative( + "fused SSA link does not support heap ownership transfer".to_string(), + )), + }) + .collect() +} + +fn lower_internal_link_args( + b: &mut FunctionBuilder, + ctx: SsaLowerCtx<'_>, + link: &FusedRegionLink, + slots: &[Option], + values: &HashMap, +) -> VmResult> { + if slots.len() != link.args.len() { + return Err(VmError::JitNative( + "fused SSA link slot count mismatch".to_string(), + )); + } + link.args + .iter() + .zip(slots) + .map(|(materialization, slot)| match materialization { + SsaMaterialization::Value(value) => values + .get(value) + .copied() + .ok_or_else(|| VmError::JitNative("fused SSA link value missing".to_string())), + SsaMaterialization::BoxInt(value) => { + let src = *values.get(value).ok_or_else(|| { + VmError::JitNative("fused SSA link int value missing".to_string()) + })?; + let addr = b.ins().stack_addr( + ctx.pointer_type, + slot.ok_or_else(|| { + VmError::JitNative("fused SSA link int slot missing".to_string()) + })?, + 0, + ); + ssa_store_int_in_value(b, ctx.layout.value, addr, src); + Ok(addr) + } + SsaMaterialization::BoxFloat(value) => { + let src = *values.get(value).ok_or_else(|| { + VmError::JitNative("fused SSA link float value missing".to_string()) + })?; + let addr = b.ins().stack_addr( + ctx.pointer_type, + slot.ok_or_else(|| { + VmError::JitNative("fused SSA link float slot missing".to_string()) + })?, + 0, + ); + ssa_store_float_in_value(b, ctx.layout.value, addr, src); + Ok(addr) + } + SsaMaterialization::BoxBool(value) => { + let src = *values.get(value).ok_or_else(|| { + VmError::JitNative("fused SSA link bool value missing".to_string()) + })?; + let addr = b.ins().stack_addr( + ctx.pointer_type, + slot.ok_or_else(|| { + VmError::JitNative("fused SSA link bool slot missing".to_string()) + })?, + 0, + ); + ssa_store_bool_in_value(b, ctx.layout.value, addr, src); + Ok(addr) + } + SsaMaterialization::BoxHeapPtr { .. } => Err(VmError::JitNative( + "fused SSA link does not support heap ownership transfer".to_string(), + )), + }) + .collect() +} + fn lower_ssa_terminator( b: &mut FunctionBuilder, - _ctx: SsaLowerCtx<'_>, + ctx: SsaLowerCtx<'_>, terminator: &SsaTerminator, values: &HashMap, block_handles: &HashMap, exit_specs: &HashMap, + internal_links: InternalRegionLinks<'_>, ) -> VmResult<()> { match terminator { SsaTerminator::Jump { target, args } => { @@ -3092,21 +4111,62 @@ fn lower_ssa_terminator( let condition = *values .get(condition) .ok_or_else(|| VmError::JitNative("SSA branch condition missing".to_string()))?; - let true_target = - ssa_branch_target_block(if_true, values, block_handles, exit_specs, false)?; - let false_target = - ssa_branch_target_block(if_false, values, block_handles, exit_specs, false)?; - let true_args = ssa_block_args(true_target.1); - let false_args = ssa_block_args(false_target.1); - b.ins().brif( - condition, - true_target.0, - &true_args, - false_target.0, - &false_args, - ); + let (true_block, true_args, true_pending) = ssa_branch_target_block( + b, + ctx, + if_true, + values, + block_handles, + exit_specs, + internal_links, + )?; + let (false_block, false_args, false_pending) = ssa_branch_target_block( + b, + ctx, + if_false, + values, + block_handles, + exit_specs, + internal_links, + )?; + let true_args = ssa_block_args(true_args); + let false_args = ssa_block_args(false_args); + b.ins() + .brif(condition, true_block, &true_args, false_block, &false_args); + if let Some(pending) = true_pending { + lower_pending_internal_link_gate(b, ctx, pending); + } + if let Some(pending) = false_pending { + lower_pending_internal_link_gate(b, ctx, pending); + } } SsaTerminator::Exit { exit } => { + if let Some((link, slots)) = internal_links.find(*exit) { + let handle = *block_handles.get(&link.child_entry).ok_or_else(|| { + VmError::JitNative("fused SSA child entry block missing".to_string()) + })?; + let child_args = lower_internal_link_args(b, ctx, link, slots, values)?; + let spec = exit_specs.get(exit).ok_or_else(|| { + VmError::JitNative("SSA internal exit lowering missing".to_string()) + })?; + let restore_args = spec + .inputs + .iter() + .map(|value| { + values.get(value).copied().ok_or_else(|| { + VmError::JitNative("SSA internal restore arg missing".to_string()) + }) + }) + .collect::>>()?; + let (target, args, pending) = + lower_internal_link_target(b, ctx, handle, child_args, spec, restore_args)?; + let args = ssa_block_args(args); + b.ins().jump(target, &args); + if let Some(pending) = pending { + lower_pending_internal_link_gate(b, ctx, pending); + } + return Ok(()); + } let spec = exit_specs .get(exit) .ok_or_else(|| VmError::JitNative("SSA exit lowering missing".to_string()))?; @@ -3163,12 +4223,18 @@ fn lower_ssa_terminator( } fn ssa_branch_target_block( + b: &mut FunctionBuilder, + ctx: SsaLowerCtx<'_>, target: &SsaBranchTarget, values: &HashMap, block_handles: &HashMap, exit_specs: &HashMap, - halted: bool, -) -> VmResult<(Block, Vec)> { + internal_links: InternalRegionLinks<'_>, +) -> VmResult<( + Block, + Vec, + Option, +)> { match target { SsaBranchTarget::Block { target, args } => { let handle = *block_handles @@ -3183,9 +4249,30 @@ fn ssa_branch_target_block( .ok_or_else(|| VmError::JitNative("SSA branch arg missing".to_string())) }) .collect::>>()?; - Ok((handle, lowered_args)) + Ok((handle, lowered_args, None)) } SsaBranchTarget::Exit(exit) => { + if let Some((link, slots)) = internal_links.find(*exit) { + let handle = *block_handles.get(&link.child_entry).ok_or_else(|| { + VmError::JitNative("fused SSA child entry block missing".to_string()) + })?; + let child_args = lower_internal_link_args(b, ctx, link, slots, values)?; + let spec = exit_specs.get(exit).ok_or_else(|| { + VmError::JitNative("SSA internal branch exit lowering missing".to_string()) + })?; + let restore_args = spec + .inputs + .iter() + .map(|value| { + values.get(value).copied().ok_or_else(|| { + VmError::JitNative( + "SSA internal branch restore arg missing".to_string(), + ) + }) + }) + .collect::>>()?; + return lower_internal_link_target(b, ctx, handle, child_args, spec, restore_args); + } let spec = exit_specs.get(exit).ok_or_else(|| { VmError::JitNative("SSA branch exit lowering missing".to_string()) })?; @@ -3199,24 +4286,108 @@ fn ssa_branch_target_block( .ok_or_else(|| VmError::JitNative("SSA exit arg missing".to_string())) }) .collect::>>()?; - Ok(( - if halted { - spec.halted_block - } else { - spec.trace_exit_block - }, - lowered_args, - )) + Ok((spec.trace_exit_block, lowered_args, None)) } } } +struct PendingInternalLinkGate { + gate_block: Block, + child_handle: Block, + child_arg_count: usize, + interrupt_block: Block, +} + +fn lower_internal_link_target( + b: &mut FunctionBuilder, + ctx: SsaLowerCtx<'_>, + child_handle: Block, + child_args: Vec, + exit_spec: &SsaExitLowering, + restore_args: Vec, +) -> VmResult<( + Block, + Vec, + Option, +)> { + let Some(_settings) = ctx.interrupt_settings else { + increment_native_region_edge_count(b, ctx); + return Ok((child_handle, child_args, None)); + }; + let gate_block = b.create_block(); + for value in child_args.iter().chain(&restore_args) { + b.append_block_param(gate_block, b.func.dfg.value_type(*value)); + } + let child_arg_count = child_args.len(); + let mut gate_args = child_args; + gate_args.extend(restore_args); + let interrupt_block = exit_spec + .interrupt_block + .ok_or_else(|| VmError::JitNative("SSA internal interrupt block missing".to_string()))?; + Ok(( + gate_block, + gate_args, + Some(PendingInternalLinkGate { + gate_block, + child_handle, + child_arg_count, + interrupt_block, + }), + )) +} + +fn lower_pending_internal_link_gate( + b: &mut FunctionBuilder, + ctx: SsaLowerCtx<'_>, + pending: PendingInternalLinkGate, +) { + let settings = ctx + .interrupt_settings + .expect("pending internal link gate requires interrupt settings"); + b.switch_to_block(pending.gate_block); + let gate_params = b.block_params(pending.gate_block).to_vec(); + let interrupt_status_block = b.create_block(); + b.append_block_param(interrupt_status_block, types::I32); + emit_interrupt_tick_inline( + b, + ctx.vm_ptr, + interrupt_status_block, + ctx.offsets, + ctx.interrupt_ops_to_advance, + settings, + ); + increment_native_region_edge_count(b, ctx); + let child_params = ssa_block_args(gate_params[..pending.child_arg_count].to_vec()); + b.ins().jump(pending.child_handle, &child_params); + + b.switch_to_block(interrupt_status_block); + let restore_params = ssa_block_args(gate_params[pending.child_arg_count..].to_vec()); + b.ins().jump(pending.interrupt_block, &restore_params); +} + +fn increment_native_region_edge_count(b: &mut FunctionBuilder, ctx: SsaLowerCtx<'_>) { + let count = b.ins().load( + types::I64, + MemFlags::new(), + ctx.vm_ptr, + ctx.offsets.jit_native_region_edge_count, + ); + let next = b.ins().iadd_imm(count, 1); + b.ins().store( + MemFlags::new(), + next, + ctx.vm_ptr, + ctx.offsets.jit_native_region_edge_count, + ); +} + fn ssa_leave_frame_status( b: &mut FunctionBuilder, pointer_type: cranelift_codegen::ir::Type, helper_refs: SsaDeoptHelperRefs, helper_addrs: SsaDeoptHelperAddrs, vm_ptr: cranelift_codegen::ir::Value, + inherited_state_ptr: cranelift_codegen::ir::Value, ret_ip: usize, ) -> VmResult { let helper_ptr = iconst_ptr_from_addr(b, pointer_type, helper_addrs.leave_frame)?; @@ -3225,25 +4396,36 @@ fn ssa_leave_frame_status( i64::try_from(ret_ip) .map_err(|_| VmError::JitNative("SSA return ip out of range".to_string()))?, ); - let call = b - .ins() - .call_indirect(helper_refs.leave_frame_ref, helper_ptr, &[vm_ptr, ret_ip]); + let call = b.ins().call_indirect( + helper_refs.leave_frame_ref, + helper_ptr, + &[vm_ptr, ret_ip, inherited_state_ptr], + ); Ok(b.inst_results(call)[0]) } +#[allow(clippy::too_many_arguments)] fn ssa_exit_action_status( b: &mut FunctionBuilder, pointer_type: cranelift_codegen::ir::Type, helper_refs: SsaDeoptHelperRefs, helper_addrs: SsaDeoptHelperAddrs, vm_ptr: cranelift_codegen::ir::Value, - exit_ip: usize, + inherited_state_ptr: cranelift_codegen::ir::Value, + exit: &crate::vm::jit::ir::SsaExit, action: SsaExitAction, ) -> VmResult { match action { - SsaExitAction::Return => { - ssa_leave_frame_status(b, pointer_type, helper_refs, helper_addrs, vm_ptr, exit_ip) - } + SsaExitAction::InterruptYield => Ok(b.ins().iconst(types::I32, STATUS_OUT_OF_FUEL as i64)), + SsaExitAction::Return => ssa_leave_frame_status( + b, + pointer_type, + helper_refs, + helper_addrs, + vm_ptr, + inherited_state_ptr, + exit.exit_ip, + ), SsaExitAction::CallValue { argc, call_ip, @@ -3266,7 +4448,7 @@ fn ssa_exit_action_status( let call = b.ins().call_indirect( helper_refs.enter_call_value_ref, helper_ptr, - &[vm_ptr, argc, call_ip, resume_ip], + &[vm_ptr, argc, call_ip, resume_ip, inherited_state_ptr], ); Ok(b.inst_results(call)[0]) } @@ -3281,12 +4463,169 @@ fn ssa_exit_action_status( ); Ok(b.inst_results(call)[0]) } else { - Ok(b.ins().iconst(types::I32, STATUS_TRACE_EXIT as i64)) + let status = crate::vm::native::encode_jit_trace_exit_status(exit.id.raw()) + .ok_or_else(|| { + VmError::JitNative("SSA exit id exceeds native status range".to_string()) + })?; + Ok(b.ins().iconst(types::I32, i64::from(status))) } } } } +fn write_inherited_state_packet( + b: &mut FunctionBuilder, + ctx: SsaLowerCtx<'_>, + exit: &crate::vm::jit::ir::SsaExit, +) -> VmResult<()> { + let inactive = b.ins().iconst(ctx.pointer_type, 0); + b.ins().store( + MemFlags::new(), + inactive, + ctx.inherited_state_ptr, + INHERITED_STATE_ACTIVE_OFFSET, + ); + b.ins().store( + MemFlags::new(), + inactive, + ctx.inherited_state_ptr, + INHERITED_STATE_DYNAMIC_TARGET_OFFSET, + ); + let entry_value_count = exit + .stack + .len() + .checked_add(exit.locals.len()) + .ok_or_else(|| VmError::JitNative("SSA inherited exit value count overflow".to_string()))?; + if entry_value_count > MAX_INHERITED_ENTRY_VALUES { + return Ok(()); + } + + let frame_key = b.ins().iconst(ctx.pointer_type, ctx.frame_key as i64); + b.ins().store( + MemFlags::new(), + frame_key, + ctx.inherited_state_ptr, + INHERITED_STATE_FRAME_KEY_OFFSET, + ); + b.ins().store( + MemFlags::new(), + ctx.active_stack_base, + ctx.inherited_state_ptr, + INHERITED_STATE_STACK_BASE_OFFSET, + ); + b.ins().store( + MemFlags::new(), + ctx.active_local_base, + ctx.inherited_state_ptr, + INHERITED_STATE_LOCAL_BASE_OFFSET, + ); + let target_ip = b.ins().iconst( + ctx.pointer_type, + i64::try_from(exit.exit_ip) + .map_err(|_| VmError::JitNative("SSA inherited target ip exceeds i64".to_string()))?, + ); + b.ins().store( + MemFlags::new(), + target_ip, + ctx.inherited_state_ptr, + INHERITED_STATE_TARGET_IP_OFFSET, + ); + let value_count = b.ins().iconst( + ctx.pointer_type, + i64::try_from(entry_value_count) + .map_err(|_| VmError::JitNative("SSA inherited value count exceeds i64".to_string()))?, + ); + b.ins().store( + MemFlags::new(), + value_count, + ctx.inherited_state_ptr, + INHERITED_STATE_VALUE_COUNT_OFFSET, + ); + let stack_ptr = b.ins().load( + ctx.pointer_type, + MemFlags::new(), + ctx.vm_ptr, + ctx.offsets.stack_ptr, + ); + let stack_base_ptr = ssa_value_addr( + b, + ctx.pointer_type, + stack_ptr, + ctx.active_stack_base, + ctx.layout.value.size, + ); + let locals_ptr = b.ins().load( + ctx.pointer_type, + MemFlags::new(), + ctx.vm_ptr, + ctx.offsets.locals_ptr, + ); + let locals_base_ptr = ssa_value_addr( + b, + ctx.pointer_type, + locals_ptr, + ctx.active_local_base, + ctx.layout.value.size, + ); + for index in 0..entry_value_count { + let value_addr = if index < exit.stack.len() { + let stack_index = b.ins().iconst( + ctx.pointer_type, + i64::try_from(index).map_err(|_| { + VmError::JitNative("SSA inherited stack index exceeds i64".to_string()) + })?, + ); + ssa_value_addr( + b, + ctx.pointer_type, + stack_base_ptr, + stack_index, + ctx.layout.value.size, + ) + } else { + let local_index = index - exit.stack.len(); + let local_index = b.ins().iconst( + ctx.pointer_type, + i64::try_from(local_index).map_err(|_| { + VmError::JitNative("SSA inherited local index exceeds i64".to_string()) + })?, + ); + ssa_value_addr( + b, + ctx.pointer_type, + locals_base_ptr, + local_index, + ctx.layout.value.size, + ) + }; + let packet_offset = index + .checked_mul(std::mem::size_of::()) + .and_then(|offset| { + usize::try_from(INHERITED_STATE_VALUES_OFFSET) + .ok() + .and_then(|base| base.checked_add(offset)) + }) + .and_then(|offset| i32::try_from(offset).ok()) + .ok_or_else(|| { + VmError::JitNative("SSA inherited packet offset exceeds i32".to_string()) + })?; + b.ins().store( + MemFlags::new(), + value_addr, + ctx.inherited_state_ptr, + packet_offset, + ); + } + let active = b.ins().iconst(ctx.pointer_type, 1); + b.ins().store( + MemFlags::new(), + active, + ctx.inherited_state_ptr, + INHERITED_STATE_ACTIVE_OFFSET, + ); + Ok(()) +} + fn lower_ssa_exit_block( b: &mut FunctionBuilder, ctx: SsaLowerCtx<'_>, @@ -3296,6 +4635,7 @@ fn lower_ssa_exit_block( ) -> VmResult<()> { let SsaLowerCtx { vm_ptr, + inherited_state_ptr, active_local_base, exit_block, pointer_type, @@ -3313,6 +4653,9 @@ fn lower_ssa_exit_block( SsaExitAction::CallValue { .. } => spec .call_value_block .ok_or_else(|| VmError::JitNative("SSA call-value exit block missing".to_string()))?, + SsaExitAction::InterruptYield => spec + .interrupt_block + .ok_or_else(|| VmError::JitNative("SSA interrupt exit block missing".to_string()))?, }; b.switch_to_block(block); let block_params = b.block_params(block).to_vec(); @@ -3399,13 +4742,17 @@ fn lower_ssa_exit_block( ); b.ins() .store(MemFlags::new(), ip_val, vm_ptr, offsets.vm_ip); + if matches!(action, SsaExitAction::TraceExit { .. }) { + write_inherited_state_packet(b, ctx, exit)?; + } let status = ssa_exit_action_status( b, pointer_type, deopt_refs, deopt_addrs, vm_ptr, - exit.exit_ip, + inherited_state_ptr, + exit, action, )?; jump_with_status(b, exit_block, status); @@ -3491,13 +4838,17 @@ fn lower_ssa_exit_block( ip_val, ], )?; + if matches!(action, SsaExitAction::TraceExit { .. }) { + write_inherited_state_packet(b, ctx, exit)?; + } let status = ssa_exit_action_status( b, pointer_type, deopt_refs, deopt_addrs, vm_ptr, - exit.exit_ip, + inherited_state_ptr, + exit, action, )?; jump_with_status(b, exit_block, status); @@ -4398,6 +5749,7 @@ struct ResolvedOffsets { fuel_ops_until_check: i32, epoch_deadline: i32, epoch_counter_ptr: i32, + jit_native_region_edge_count: i32, } fn emit_interrupt_tick_inline( @@ -4620,11 +5972,13 @@ fn resolve_offsets(layout: NativeStackLayout) -> VmResult { fuel_ops_until_check: layout.vm_fuel_ops_until_check_offset, epoch_deadline: layout.vm_epoch_deadline_offset, epoch_counter_ptr: layout.vm_epoch_counter_ptr_offset, + jit_native_region_edge_count: layout.vm_jit_native_region_edge_count_offset, }) } pub(crate) fn compile_trace( trace: &JitTrace, + internal_links: &[FusedRegionLink], interrupt_settings: Option, profile: NativeCompileProfile, drop_contract_events_enabled: bool, @@ -4633,9 +5987,10 @@ pub(crate) fn compile_trace( return Err(VmError::InvalidFuelCheckInterval(0)); } - try_compile_ssa_trace( + let body = try_compile_ssa_trace( trace, &trace.ssa, + internal_links, interrupt_settings, profile, drop_contract_events_enabled, @@ -4645,9 +6000,40 @@ pub(crate) fn compile_trace( "SSA native lowering does not support trace {} at root_ip {}", trace.id, trace.root_ip )) + })?; + let tail_entry = body.tail_entry; + let lowering_kind = body.lowering_kind; + let mut code = body.code; + let mut wrapper = compile_system_inherited_tail_wrapper(tail_entry)?; + code.extend_from_slice(&wrapper.code); + wrapper._keepalive.dependencies.push(body.keepalive); + Ok(CompiledTrace { + entry: wrapper.entry, + tail_entry, + keepalive: wrapper._keepalive, + code, + lowering_kind, }) } +fn native_tail_isa() -> VmResult { + let cached = CRANELIFT_TAIL_ISA.get_or_init(|| { + let mut flag_builder = settings::builder(); + flag_builder + .set("opt_level", "speed") + .map_err(|err| format!("failed to set cranelift opt_level: {err}"))?; + flag_builder + .set("preserve_frame_pointers", "true") + .map_err(|err| format!("failed to preserve tail-call frame pointers: {err}"))?; + let isa_builder = cranelift_native::builder() + .map_err(|err| format!("failed to build native tail ISA: {err}"))?; + isa_builder + .finish(settings::Flags::new(flag_builder)) + .map_err(|err| format!("failed to finalize cranelift tail ISA: {err}")) + }); + cached.clone().map_err(VmError::JitNative) +} + fn native_isa(profile: NativeCompileProfile) -> VmResult { let cached = match profile { NativeCompileProfile::Jit => &CRANELIFT_JIT_ISA, diff --git a/src/vm/jit/native/mod.rs b/src/vm/jit/native/mod.rs index 4cbfe447..648517cd 100644 --- a/src/vm/jit/native/mod.rs +++ b/src/vm/jit/native/mod.rs @@ -1,12 +1,15 @@ #![allow(dead_code)] -#[cfg(not(feature = "cranelift-jit"))] -use super::super::super::VmError; -use super::super::super::VmResult; +use super::super::super::{VmError, VmResult}; +use super::ir::{SsaMaterialization, SsaValueRepr}; +use crate::ValueType; +use std::collections::HashMap; +use std::sync::Arc; +use std::sync::atomic::{AtomicPtr, Ordering}; pub(crate) use crate::vm::native::{ NativeInterruptSettings, STATUS_CONTINUE, STATUS_ERROR, STATUS_HALTED, STATUS_LINKED_CONTINUE, STATUS_OUT_OF_FUEL, STATUS_TRACE_EXIT, STATUS_WAITING, STATUS_YIELDED, clear_bridge_error, - selected_codegen_backend, store_bridge_error, take_bridge_error, + decode_jit_trace_exit_status, selected_codegen_backend, store_bridge_error, take_bridge_error, }; #[cfg(feature = "cranelift-jit")] @@ -30,6 +33,146 @@ impl TraceLoweringKind { } } +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum SideEntryOwnership { + Borrowed, + Owned, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum InheritedStateAbiClass { + ScalarInt, + ScalarFloat, + ScalarBool, + HeapPointer { + tag: ValueType, + ownership: SideEntryOwnership, + }, + Tagged(SideEntryOwnership), +} + +pub(crate) fn classify_side_entry_repr( + repr: SsaValueRepr, + ownership: SideEntryOwnership, +) -> InheritedStateAbiClass { + match repr { + SsaValueRepr::I64 => InheritedStateAbiClass::ScalarInt, + SsaValueRepr::F64 => InheritedStateAbiClass::ScalarFloat, + SsaValueRepr::Bool => InheritedStateAbiClass::ScalarBool, + SsaValueRepr::HeapPtr(tag) => InheritedStateAbiClass::HeapPointer { tag, ownership }, + SsaValueRepr::Tagged => InheritedStateAbiClass::Tagged(ownership), + } +} + +pub(crate) fn classify_side_entry_materialization( + materialization: &SsaMaterialization, + defining_repr: Option, +) -> Option { + match materialization { + SsaMaterialization::Value(_) => { + defining_repr.map(|repr| classify_side_entry_repr(repr, SideEntryOwnership::Borrowed)) + } + SsaMaterialization::BoxInt(_) => Some(InheritedStateAbiClass::ScalarInt), + SsaMaterialization::BoxFloat(_) => Some(InheritedStateAbiClass::ScalarFloat), + SsaMaterialization::BoxBool(_) => Some(InheritedStateAbiClass::ScalarBool), + SsaMaterialization::BoxHeapPtr { tag, .. } => Some(InheritedStateAbiClass::HeapPointer { + tag: *tag, + ownership: SideEntryOwnership::Owned, + }), + } +} + +pub(crate) struct NativeSideLinkSlot { + entry: AtomicPtr, +} + +impl NativeSideLinkSlot { + pub(crate) const fn new() -> Self { + Self { + entry: AtomicPtr::new(std::ptr::null_mut()), + } + } + + pub(crate) fn target(&self) -> *mut u8 { + self.entry.load(Ordering::Acquire) + } + + pub(crate) fn publish(&self, entry: *const u8) { + self.entry.store(entry.cast_mut(), Ordering::Release); + } + + pub(crate) fn clear(&self) { + self.entry.store(std::ptr::null_mut(), Ordering::Release); + } + + pub(crate) fn address(self: &Arc) -> *mut *mut u8 { + self.entry.as_ptr() + } +} + +impl Default for NativeSideLinkSlot { + fn default() -> Self { + Self::new() + } +} + +pub(crate) const CONTINUE_SLOT_ID: u32 = u32::MAX - 1; + +pub(crate) struct CompiledTraceDispatcher { + pub(crate) entry: *const u8, + pub(crate) tail_entry: *const u8, + pub(crate) keepalives: Vec, + pub(crate) code: Vec, + pub(crate) slots: HashMap>, +} + +#[cfg(feature = "cranelift-jit")] +pub(crate) fn compile_native_trace_dispatcher( + trace_id: usize, + trace_entry: *const u8, + trace: &super::JitTrace, +) -> VmResult { + let mut slots = HashMap::new(); + let mut descriptors = Vec::with_capacity(trace.ssa.exits.len()); + for exit in &trace.ssa.exits { + let status = + crate::vm::native::encode_jit_trace_exit_status(exit.id.raw()).ok_or_else(|| { + VmError::JitNative("SSA exit id exceeds native status range".to_string()) + })?; + let slot = Arc::new(NativeSideLinkSlot::new()); + descriptors.push((status, slot.address() as usize)); + slots.insert(exit.id.raw(), slot); + } + let slot = Arc::new(NativeSideLinkSlot::new()); + descriptors.push((STATUS_CONTINUE, slot.address() as usize)); + slots.insert(CONTINUE_SLOT_ID, slot); + let dispatcher = lower::compile_tail_trace_dispatcher(trace_entry, trace_id, &descriptors)?; + let tail_entry = dispatcher.entry(); + let wrapper = lower::compile_system_inherited_tail_wrapper(tail_entry)?; + let (tail_entry, dispatcher_keepalive, dispatcher_code) = dispatcher.into_parts(); + let (entry, wrapper_keepalive, wrapper_code) = wrapper.into_parts(); + let mut code = dispatcher_code; + code.extend_from_slice(&wrapper_code); + Ok(CompiledTraceDispatcher { + entry, + tail_entry, + keepalives: vec![dispatcher_keepalive, wrapper_keepalive], + code, + slots, + }) +} + +#[cfg(not(feature = "cranelift-jit"))] +pub(crate) fn compile_native_trace_dispatcher( + _trace_id: usize, + _trace_entry: *const u8, + _trace: &super::JitTrace, +) -> VmResult { + Err(VmError::JitNative( + "native JIT backend is disabled (feature 'cranelift-jit' is not enabled)".to_string(), + )) +} + #[cfg(feature = "cranelift-jit")] pub(crate) use lower::{CompiledTrace, TraceKeepAlive}; @@ -39,6 +182,7 @@ pub(crate) struct TraceKeepAlive; #[cfg(not(feature = "cranelift-jit"))] pub(crate) struct CompiledTrace { pub entry: *const u8, + pub tail_entry: *const u8, pub code: Vec, pub keepalive: TraceKeepAlive, pub lowering_kind: TraceLoweringKind, @@ -53,6 +197,7 @@ pub(super) fn compile_native_trace( ) -> VmResult> { Ok(Box::new(lower::compile_trace( trace, + &[], interrupt_settings, profile, drop_contract_events_enabled, @@ -71,9 +216,225 @@ pub(super) fn compile_native_trace( )) } +#[cfg(feature = "cranelift-jit")] +pub(super) fn compile_native_region( + region: &super::region::FusedRegion, + interrupt_settings: Option, + profile: NativeCompileProfile, + drop_contract_events_enabled: bool, +) -> VmResult> { + Ok(Box::new(lower::compile_trace( + ®ion.trace, + ®ion.links, + interrupt_settings, + profile, + drop_contract_events_enabled, + )?)) +} + +#[cfg(not(feature = "cranelift-jit"))] +pub(super) fn compile_native_region( + _region: &super::region::FusedRegion, + _interrupt_settings: Option, + _profile: NativeCompileProfile, + _drop_contract_events_enabled: bool, +) -> VmResult> { + Err(VmError::JitNative( + "native JIT backend is disabled (feature 'cranelift-jit' is not enabled)".to_string(), + )) +} + #[cfg(test)] mod tests { - use super::selected_codegen_backend; + use super::lower::{ + compile_system_owned_tail_wrapper, compile_system_tail_wrapper, + compile_tail_owned_clear_body, compile_tail_owned_side_link_body, + compile_tail_side_link_body, compile_tail_status_body, + }; + use super::{ + InheritedStateAbiClass, NativeSideLinkSlot, SideEntryOwnership, + classify_side_entry_materialization, classify_side_entry_repr, selected_codegen_backend, + }; + use crate::vm::jit::ir::{SsaMaterialization, SsaValueId, SsaValueRepr}; + use crate::{Value, ValueType}; + use std::sync::Arc; + + #[cfg(target_os = "linux")] + fn assert_executable_mapping_is_not_writable(entry: *const u8) { + let address = entry as usize; + let maps = std::fs::read_to_string("/proc/self/maps").expect("process maps should read"); + let mapping = maps + .lines() + .find(|line| { + let Some((range, _)) = line.split_once(' ') else { + return false; + }; + let Some((start, end)) = range.split_once('-') else { + return false; + }; + let Ok(start) = usize::from_str_radix(start, 16) else { + return false; + }; + let Ok(end) = usize::from_str_radix(end, 16) else { + return false; + }; + start <= address && address < end + }) + .expect("entry mapping should exist"); + let permissions = mapping + .split_whitespace() + .nth(1) + .expect("mapping permissions should exist"); + assert!(permissions.contains('x'), "{mapping}"); + assert!(!permissions.contains('w'), "{mapping}"); + } + + #[test] + fn side_entry_abi_classifies_scalar_pointer_tagged_borrowed_and_owned_values() { + let value = SsaValueId::new(7); + assert_eq!( + classify_side_entry_materialization(&SsaMaterialization::BoxInt(value), None), + Some(InheritedStateAbiClass::ScalarInt) + ); + assert_eq!( + classify_side_entry_materialization(&SsaMaterialization::BoxFloat(value), None), + Some(InheritedStateAbiClass::ScalarFloat) + ); + assert_eq!( + classify_side_entry_materialization(&SsaMaterialization::BoxBool(value), None), + Some(InheritedStateAbiClass::ScalarBool) + ); + for (repr, expected) in [ + (SsaValueRepr::I64, InheritedStateAbiClass::ScalarInt), + (SsaValueRepr::F64, InheritedStateAbiClass::ScalarFloat), + (SsaValueRepr::Bool, InheritedStateAbiClass::ScalarBool), + ( + SsaValueRepr::HeapPtr(ValueType::Array), + InheritedStateAbiClass::HeapPointer { + tag: ValueType::Array, + ownership: SideEntryOwnership::Borrowed, + }, + ), + ( + SsaValueRepr::Tagged, + InheritedStateAbiClass::Tagged(SideEntryOwnership::Borrowed), + ), + ] { + assert_eq!( + classify_side_entry_materialization(&SsaMaterialization::Value(value), Some(repr),), + Some(expected) + ); + } + assert_eq!( + classify_side_entry_materialization(&SsaMaterialization::Value(value), None), + None + ); + assert_eq!( + classify_side_entry_materialization( + &SsaMaterialization::BoxHeapPtr { + value, + tag: ValueType::String, + }, + None, + ), + Some(InheritedStateAbiClass::HeapPointer { + tag: ValueType::String, + ownership: SideEntryOwnership::Owned, + }) + ); + assert_eq!( + classify_side_entry_repr( + SsaValueRepr::HeapPtr(ValueType::Array), + SideEntryOwnership::Borrowed, + ), + InheritedStateAbiClass::HeapPointer { + tag: ValueType::Array, + ownership: SideEntryOwnership::Borrowed, + } + ); + } + + #[cfg(feature = "cranelift-jit")] + #[test] + fn trace_jit_side_link_slot_switches_between_deopt_and_child() { + if selected_codegen_backend() != "native" { + return; + } + const DEOPT_STATUS: i32 = 17; + const CHILD_STATUS: i32 = 23; + let slot = Arc::new(NativeSideLinkSlot::new()); + let child = compile_tail_status_body(CHILD_STATUS).expect("tail child should compile"); + let root = compile_tail_side_link_body(slot.address() as usize, DEOPT_STATUS) + .expect("tail root should compile"); + let wrapper = + compile_system_tail_wrapper(root.entry()).expect("system wrapper should compile"); + #[cfg(target_os = "linux")] + assert_executable_mapping_is_not_writable(wrapper.entry()); + assert!(child.code_len() > 0); + assert!(root.code_len() > 0); + assert!(wrapper.code_len() > 0); + let entry = unsafe { + std::mem::transmute::<*const u8, unsafe extern "C" fn(*mut crate::Vm) -> i32>( + wrapper.entry(), + ) + }; + + assert!(slot.target().is_null()); + assert_eq!(unsafe { entry(std::ptr::null_mut()) }, DEOPT_STATUS); + slot.publish(child.entry()); + assert_eq!(slot.target().cast_const(), child.entry()); + assert_eq!(unsafe { entry(std::ptr::null_mut()) }, CHILD_STATUS); + slot.clear(); + assert!(slot.target().is_null()); + assert_eq!(unsafe { entry(std::ptr::null_mut()) }, DEOPT_STATUS); + } + + #[cfg(feature = "cranelift-jit")] + #[test] + fn trace_jit_side_entry_transfers_owned_values_once() { + if selected_codegen_backend() != "native" { + return; + } + const DEOPT_STATUS: i32 = 29; + const CHILD_STATUS: i32 = 31; + let slot = Arc::new(NativeSideLinkSlot::new()); + let child = + compile_tail_owned_clear_body(CHILD_STATUS).expect("owned child should compile"); + let root = compile_tail_owned_side_link_body(slot.address() as usize, DEOPT_STATUS) + .expect("owned root should compile"); + let wrapper = compile_system_owned_tail_wrapper(root.entry()) + .expect("owned system wrapper should compile"); + let entry = unsafe { + std::mem::transmute::<*const u8, unsafe extern "C" fn(*mut crate::Vm, *mut Value) -> i32>( + wrapper.entry(), + ) + }; + let backing = Arc::new("owned".to_string()); + let mut owned = Value::String(backing.clone()); + assert_eq!(Arc::strong_count(&backing), 2); + + assert_eq!( + unsafe { entry(std::ptr::null_mut(), &mut owned) }, + DEOPT_STATUS + ); + assert!(matches!(owned, Value::String(_))); + assert_eq!(Arc::strong_count(&backing), 2); + + slot.publish(child.entry()); + assert_eq!( + unsafe { entry(std::ptr::null_mut(), &mut owned) }, + CHILD_STATUS + ); + assert!(matches!(owned, Value::Null)); + assert_eq!(Arc::strong_count(&backing), 1); + + assert_eq!( + unsafe { entry(std::ptr::null_mut(), &mut owned) }, + CHILD_STATUS + ); + assert!(matches!(owned, Value::Null)); + assert_eq!(Arc::strong_count(&backing), 1); + } #[test] fn selected_backend_is_native() { diff --git a/src/vm/jit/recorder.rs b/src/vm/jit/recorder.rs index 23636367..ce1dae97 100644 --- a/src/vm/jit/recorder.rs +++ b/src/vm/jit/recorder.rs @@ -512,7 +512,9 @@ enum SpecializedBuiltinKind { StringConcat, BytesConcat, BytesFromArrayU8, + BytesToUtf8Ascii, BytesToArrayU8, + ArrayNew, ArrayLen, ArrayGet, ArrayHas, @@ -1225,6 +1227,19 @@ pub(crate) fn record_trace_with_local_count( argc, yields, } => { + if builtin == Some(BuiltinFunction::ArrayNew) && argc == 0 { + let (name, out) = emit_specialized_builtin_call( + &mut builder, + current_block, + ip, + &mut frame, + SpecializedBuiltinKind::ArrayNew, + )?; + op_names.push(name.to_string()); + frame.push(out); + has_useful_native_computation = true; + continue; + } if let Some(builtin) = builtin && argc > 0 { @@ -1277,11 +1292,17 @@ pub(crate) fn record_trace_with_local_count( .get(usize::from(index)) .map(|import| import.return_type) .unwrap_or(ValueType::Unknown); + let (return_repr, return_info) = match return_type { + ValueType::Int => (SsaValueRepr::I64, ValueInfo::int(None)), + ValueType::Float => (SsaValueRepr::F64, ValueInfo::float(None)), + ValueType::Bool => (SsaValueRepr::Bool, ValueInfo::bool(None)), + _ => (SsaValueRepr::Tagged, ValueInfo::tagged_typed(return_type)), + }; let value = builder .append_value_inst( current_block, ip, - SsaValueRepr::Tagged, + return_repr, SsaInstKind::HostCall { import: index, args, @@ -1290,7 +1311,7 @@ pub(crate) fn record_trace_with_local_count( .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; frame.push(SymbolicValue { value, - info: ValueInfo::tagged_typed(return_type), + info: return_info, }); has_call = true; op_names.push("host_call".to_string()); @@ -1471,6 +1492,13 @@ fn infer_loop_header_plan( .. } => { if argc == 0 { + if builtin == BuiltinFunction::ArrayNew { + let _ = analyze_specialized_builtin_call( + &mut frame, + SpecializedBuiltinKind::ArrayNew, + )?; + continue; + } return Ok(None); } let args = call_arg_slice(&frame.stack, usize::from(argc))?; @@ -1508,7 +1536,12 @@ fn infer_loop_header_plan( .get(usize::from(index)) .map(|import| import.return_type) .unwrap_or(ValueType::Unknown); - frame.push(ValueInfo::tagged_typed(return_type)); + frame.push(match return_type { + ValueType::Int => ValueInfo::int(None), + ValueType::Float => ValueInfo::float(None), + ValueType::Bool => ValueInfo::bool(None), + _ => ValueInfo::tagged_typed(return_type), + }); } DecodedOp::Call { .. } | DecodedOp::CallValue { .. } => return Ok(None), DecodedOp::Brfalse { @@ -2771,6 +2804,9 @@ fn select_specialized_builtin_kind( BuiltinFunction::StringSplitLiteral => { return Some(SpecializedBuiltinKind::StringSplitLiteral); } + BuiltinFunction::BytesToUtf8 => { + return Some(SpecializedBuiltinKind::BytesToUtf8Ascii); + } BuiltinFunction::MapIterNext => return Some(SpecializedBuiltinKind::MapIterNext), BuiltinFunction::MapIterTakeKey => return Some(SpecializedBuiltinKind::MapIterTakeKey), BuiltinFunction::MapIterTakeValue => { @@ -2976,11 +3012,20 @@ fn analyze_specialized_builtin_call( frame.push(ValueInfo::tagged_typed(ValueType::Bytes)); Ok("bytes_from_array_u8") } + SpecializedBuiltinKind::BytesToUtf8Ascii => { + let _ = frame.pop()?; + frame.push(ValueInfo::tagged_typed(ValueType::String)); + Ok("bytes_to_utf8_ascii") + } SpecializedBuiltinKind::BytesToArrayU8 => { let _ = frame.pop()?; frame.push(ValueInfo::tagged_typed(ValueType::Array)); Ok("bytes_to_array_u8") } + SpecializedBuiltinKind::ArrayNew => { + frame.push(ValueInfo::tagged_typed(ValueType::Array)); + Ok("array_new") + } SpecializedBuiltinKind::ArrayLen => { let _ = frame.pop()?; frame.push(ValueInfo::int(None)); @@ -3562,6 +3607,30 @@ fn emit_specialized_builtin_call( .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; Ok(("bytes_from_array_u8", out)) } + SpecializedBuiltinKind::BytesToUtf8Ascii => { + let bytes = ensure_heap_ptr( + builder, + block, + ip, + frame.pop()?, + HeapContainerKind::Bytes.value_type(), + )?; + let out = builder + .append_value_inst( + block, + ip, + SsaValueRepr::Tagged, + SsaInstKind::BytesToUtf8Ascii { + bytes: bytes.value.id, + }, + ) + .map(|value| SymbolicValue { + value, + info: ValueInfo::tagged_typed(ValueType::String), + }) + .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; + Ok(("bytes_to_utf8_ascii", out)) + } SpecializedBuiltinKind::BytesToArrayU8 => { let bytes = ensure_heap_ptr( builder, @@ -3586,6 +3655,16 @@ fn emit_specialized_builtin_call( .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; Ok(("bytes_to_array_u8", out)) } + SpecializedBuiltinKind::ArrayNew => { + let out = builder + .append_value_inst(block, ip, SsaValueRepr::Tagged, SsaInstKind::ArrayNew) + .map(|value| SymbolicValue { + value, + info: ValueInfo::tagged_typed(ValueType::Array), + }) + .map_err(|err| TraceRecordError::InvalidIr(err.to_string()))?; + Ok(("array_new", out)) + } SpecializedBuiltinKind::ArrayLen => { let array = frame.pop()?; let array = ensure_heap_ptr( diff --git a/src/vm/jit/region.rs b/src/vm/jit/region.rs new file mode 100644 index 00000000..e23fbbd4 --- /dev/null +++ b/src/vm/jit/region.rs @@ -0,0 +1,579 @@ +use std::collections::HashMap; + +use crate::{VmError, VmResult}; + +use super::JitTrace; +use super::deopt::SideTraceImport; +use super::ir::{ + SsaBlock, SsaBlockId, SsaBranchTarget, SsaExit, SsaExitId, SsaInst, SsaInstKind, + SsaMaterialization, SsaTerminator, SsaTrace, SsaValueId, +}; +use super::trace::TraceExitKey; + +#[derive(Clone, Debug)] +#[cfg_attr(not(feature = "cranelift-jit"), allow(dead_code))] +pub(crate) struct FusedRegionLink { + pub(crate) exit: SsaExitId, + pub(crate) child_entry: SsaBlockId, + pub(crate) args: Vec, +} + +#[derive(Clone, Debug)] +pub(crate) struct FusedRegion { + pub(crate) trace: JitTrace, + #[cfg_attr(not(feature = "cranelift-jit"), allow(dead_code))] + pub(crate) links: Vec, + pub(crate) exit_keys: HashMap, +} + +pub(crate) fn fuse_two_trace_region( + parent: &JitTrace, + child: &JitTrace, + import: &SideTraceImport, + back_import: Option<&SideTraceImport>, +) -> VmResult { + let value_offset = parent + .ssa + .blocks + .iter() + .flat_map(|block| { + block.params.iter().map(|param| param.value.id).chain( + block + .insts + .iter() + .filter_map(|inst| inst.output.map(|out| out.id)), + ) + }) + .map(SsaValueId::raw) + .max() + .map_or(0, |max| max.saturating_add(1)); + let block_offset = u32::try_from(parent.ssa.blocks.len()) + .map_err(|_| VmError::JitNative("region parent block count exceeds u32".to_string()))?; + let exit_offset = u32::try_from(parent.ssa.exits.len()) + .map_err(|_| VmError::JitNative("region parent exit count exceeds u32".to_string()))?; + + let mut blocks = parent.ssa.blocks.clone(); + blocks.extend( + child + .ssa + .blocks + .iter() + .cloned() + .map(|block| offset_block(block, value_offset, block_offset, exit_offset)) + .collect::>>()?, + ); + let mut exits = parent.ssa.exits.clone(); + if let Some(back_import) = back_import { + for exit in &mut exits { + merge_dirty_locals(&mut exit.dirty_locals, &back_import.dirty_locals); + } + } + exits.extend( + child + .ssa + .exits + .iter() + .cloned() + .map(|mut exit| { + merge_dirty_locals(&mut exit.dirty_locals, &import.dirty_locals); + offset_exit(exit, value_offset, exit_offset) + }) + .collect::>>()?, + ); + + let child_entry = offset_block_id(child.ssa.entry, block_offset)?; + let ssa = SsaTrace { + root_ip: parent.ssa.root_ip, + entry_stack_depth: parent.ssa.entry_stack_depth, + entry: parent.ssa.entry, + blocks, + exits, + }; + ssa.verify().map_err(|err| { + VmError::JitNative(format!( + "fused two-trace region failed SSA verification: {err:?}" + )) + })?; + + let mut trace = parent.clone(); + trace.ssa = ssa; + trace.has_call |= child.has_call; + trace.has_yielding_call |= child.has_yielding_call; + trace.op_names.extend(child.op_names.iter().cloned()); + + let mut exit_keys = HashMap::new(); + for exit in &parent.ssa.exits { + exit_keys.insert( + exit.id.raw(), + TraceExitKey { + parent_trace_id: parent.id, + exit_id: exit.id, + }, + ); + } + for exit in &child.ssa.exits { + let fused_exit = offset_exit_id(exit.id, exit_offset)?; + exit_keys.insert( + fused_exit.raw(), + TraceExitKey { + parent_trace_id: child.id, + exit_id: exit.id, + }, + ); + } + + let mut links = vec![FusedRegionLink { + exit: import.parent_exit, + child_entry, + args: import.args.clone(), + }]; + if let Some(back_import) = back_import { + let mut args = back_import.args.clone(); + for materialization in &mut args { + offset_materialization(materialization, value_offset)?; + } + links.push(FusedRegionLink { + exit: offset_exit_id(back_import.parent_exit, exit_offset)?, + child_entry: parent.ssa.entry, + args, + }); + } + + Ok(FusedRegion { + trace, + links, + exit_keys, + }) +} + +fn merge_dirty_locals(target: &mut Vec, incoming: &[bool]) { + if target.len() < incoming.len() { + target.resize(incoming.len(), false); + } + for (target, incoming) in target.iter_mut().zip(incoming) { + *target |= *incoming; + } +} + +fn offset_block( + mut block: SsaBlock, + value_offset: u32, + block_offset: u32, + exit_offset: u32, +) -> VmResult { + block.id = offset_block_id(block.id, block_offset)?; + for param in &mut block.params { + param.value.id = offset_value_id(param.value.id, value_offset)?; + } + for inst in &mut block.insts { + offset_inst(inst, value_offset)?; + } + if let Some(terminator) = &mut block.terminator { + offset_terminator(terminator, value_offset, block_offset, exit_offset)?; + } + Ok(block) +} + +fn offset_exit(mut exit: SsaExit, value_offset: u32, exit_offset: u32) -> VmResult { + exit.id = offset_exit_id(exit.id, exit_offset)?; + for materialization in exit.stack.iter_mut().chain(&mut exit.locals) { + offset_materialization(materialization, value_offset)?; + } + Ok(exit) +} + +fn offset_inst(inst: &mut SsaInst, value_offset: u32) -> VmResult<()> { + if let Some(output) = &mut inst.output { + output.id = offset_value_id(output.id, value_offset)?; + } + remap_inst_inputs(&mut inst.kind, |value| offset_value_id(value, value_offset)) +} + +fn remap_inst_inputs( + kind: &mut SsaInstKind, + mut remap: impl FnMut(SsaValueId) -> VmResult, +) -> VmResult<()> { + macro_rules! one { + ($field:expr) => {{ + *$field = remap(*$field)?; + }}; + } + macro_rules! two { + ($lhs:expr, $rhs:expr) => {{ + one!($lhs); + one!($rhs); + }}; + } + match kind { + SsaInstKind::Constant(_) | SsaInstKind::ArrayNew => {} + SsaInstKind::HostCall { args, .. } => { + for arg in args { + one!(arg); + } + } + SsaInstKind::UnboxInt { input } + | SsaInstKind::UnboxFloat { input } + | SsaInstKind::UnboxBool { input } + | SsaInstKind::UnboxHeapPtr { input, .. } + | SsaInstKind::ValueLen { value: input } + | SsaInstKind::StringLen { text: input } + | SsaInstKind::BytesLen { bytes: input } + | SsaInstKind::ArrayLen { array: input } + | SsaInstKind::MapLen { map: input } + | SsaInstKind::IntNeg { input } + | SsaInstKind::BoolNot { input } + | SsaInstKind::FloatNeg { input } + | SsaInstKind::StringLowerAscii { text: input } + | SsaInstKind::TypeOf { value: input } + | SsaInstKind::ToString { value: input } + | SsaInstKind::BytesFromArrayU8 { array: input } + | SsaInstKind::BytesToUtf8Ascii { bytes: input } + | SsaInstKind::BytesToArrayU8 { bytes: input } + | SsaInstKind::MapIterNext { slot: input } + | SsaInstKind::MapIterTakeKey { slot: input } + | SsaInstKind::MapIterTakeValue { slot: input } => one!(input), + SsaInstKind::StringSlice { + text, + start, + length, + } + | SsaInstKind::BytesSlice { + bytes: text, + start, + length, + } => { + one!(text); + one!(start); + one!(length); + } + SsaInstKind::RegexReplace { + pattern, + text, + replacement, + } + | SsaInstKind::StringReplaceLiteral { + text, + needle: pattern, + replacement, + } => { + one!(pattern); + one!(text); + one!(replacement); + } + SsaInstKind::ArraySet { + array, + index, + value, + } + | SsaInstKind::MapSet { + map: array, + key: index, + value, + } => { + one!(array); + one!(index); + one!(value); + } + SsaInstKind::StringGet { text, index } + | SsaInstKind::BytesGet { bytes: text, index } + | SsaInstKind::BytesHas { bytes: text, index } + | SsaInstKind::StringContains { + text, + needle: index, + } + | SsaInstKind::RegexMatch { + pattern: text, + text: index, + } + | SsaInstKind::StringSplitLiteral { + text, + delimiter: index, + } + | SsaInstKind::StringConcat { + lhs: text, + rhs: index, + } + | SsaInstKind::BytesConcat { + lhs: text, + rhs: index, + } + | SsaInstKind::ArrayGet { array: text, index } + | SsaInstKind::ArrayHas { array: text, index } + | SsaInstKind::ArrayPush { + array: text, + value: index, + } + | SsaInstKind::MapGet { + map: text, + key: index, + } + | SsaInstKind::MapHas { + map: text, + key: index, + } + | SsaInstKind::IntAdd { + lhs: text, + rhs: index, + } + | SsaInstKind::IntSub { + lhs: text, + rhs: index, + } + | SsaInstKind::IntMul { + lhs: text, + rhs: index, + } + | SsaInstKind::IntDiv { + lhs: text, + rhs: index, + } + | SsaInstKind::IntMod { + lhs: text, + rhs: index, + } + | SsaInstKind::IntShl { + lhs: text, + rhs: index, + } + | SsaInstKind::IntShr { + lhs: text, + rhs: index, + } + | SsaInstKind::IntLshr { + lhs: text, + rhs: index, + } + | SsaInstKind::BoolAnd { + lhs: text, + rhs: index, + } + | SsaInstKind::BoolOr { + lhs: text, + rhs: index, + } + | SsaInstKind::FloatAdd { + lhs: text, + rhs: index, + } + | SsaInstKind::FloatSub { + lhs: text, + rhs: index, + } + | SsaInstKind::FloatMul { + lhs: text, + rhs: index, + } + | SsaInstKind::FloatDiv { + lhs: text, + rhs: index, + } + | SsaInstKind::FloatMod { + lhs: text, + rhs: index, + } + | SsaInstKind::FloatCmpEq { + lhs: text, + rhs: index, + } + | SsaInstKind::FloatCmpLt { + lhs: text, + rhs: index, + } + | SsaInstKind::FloatCmpGt { + lhs: text, + rhs: index, + } + | SsaInstKind::IntCmpEq { + lhs: text, + rhs: index, + } + | SsaInstKind::ValueCmpEq { + lhs: text, + rhs: index, + } + | SsaInstKind::IntCmpLt { + lhs: text, + rhs: index, + } + | SsaInstKind::IntCmpGt { + lhs: text, + rhs: index, + } => two!(text, index), + SsaInstKind::IntAddImm { lhs, .. } + | SsaInstKind::IntSubImm { lhs, .. } + | SsaInstKind::IntMulImm { lhs, .. } + | SsaInstKind::IntDivImm { lhs, .. } + | SsaInstKind::IntModImm { lhs, .. } + | SsaInstKind::IntShlImm { lhs, .. } + | SsaInstKind::IntShrImm { lhs, .. } + | SsaInstKind::IntLshrImm { lhs, .. } + | SsaInstKind::IntCmpLtImm { lhs, .. } + | SsaInstKind::IntCmpGtImm { lhs, .. } => one!(lhs), + } + Ok(()) +} + +fn offset_terminator( + terminator: &mut SsaTerminator, + value_offset: u32, + block_offset: u32, + exit_offset: u32, +) -> VmResult<()> { + match terminator { + SsaTerminator::Jump { target, args } => { + *target = offset_block_id(*target, block_offset)?; + for arg in args { + *arg = offset_value_id(*arg, value_offset)?; + } + } + SsaTerminator::BranchBool { + condition, + if_true, + if_false, + } => { + *condition = offset_value_id(*condition, value_offset)?; + offset_branch_target(if_true, value_offset, block_offset, exit_offset)?; + offset_branch_target(if_false, value_offset, block_offset, exit_offset)?; + } + SsaTerminator::Exit { exit } + | SsaTerminator::Return { exit } + | SsaTerminator::CallValue { exit, .. } => { + *exit = offset_exit_id(*exit, exit_offset)?; + } + } + Ok(()) +} + +fn offset_branch_target( + target: &mut SsaBranchTarget, + value_offset: u32, + block_offset: u32, + exit_offset: u32, +) -> VmResult<()> { + match target { + SsaBranchTarget::Block { target, args } => { + *target = offset_block_id(*target, block_offset)?; + for arg in args { + *arg = offset_value_id(*arg, value_offset)?; + } + } + SsaBranchTarget::Exit(exit) => *exit = offset_exit_id(*exit, exit_offset)?, + } + Ok(()) +} + +fn offset_materialization( + materialization: &mut SsaMaterialization, + value_offset: u32, +) -> VmResult<()> { + let value = match materialization { + SsaMaterialization::Value(value) + | SsaMaterialization::BoxInt(value) + | SsaMaterialization::BoxFloat(value) + | SsaMaterialization::BoxBool(value) + | SsaMaterialization::BoxHeapPtr { value, .. } => value, + }; + *value = offset_value_id(*value, value_offset)?; + Ok(()) +} + +fn offset_value_id(value: SsaValueId, offset: u32) -> VmResult { + value + .raw() + .checked_add(offset) + .map(SsaValueId::new) + .ok_or_else(|| VmError::JitNative("region SSA value id overflow".to_string())) +} + +fn offset_block_id(block: SsaBlockId, offset: u32) -> VmResult { + u32::try_from(block.index()) + .ok() + .and_then(|raw| raw.checked_add(offset)) + .map(SsaBlockId::new) + .ok_or_else(|| VmError::JitNative("region SSA block id overflow".to_string())) +} + +fn offset_exit_id(exit: SsaExitId, offset: u32) -> VmResult { + exit.raw() + .checked_add(offset) + .map(SsaExitId::new) + .ok_or_else(|| VmError::JitNative("region SSA exit id overflow".to_string())) +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::vm::jit::JitTraceTerminal; + use crate::vm::jit::ir::{SsaTerminator, SsaTraceBuilder}; + + fn test_trace(id: usize, root_ip: usize, exit_ip: usize) -> (JitTrace, SsaExitId) { + let mut ssa = SsaTraceBuilder::new(root_ip, 0); + let entry = ssa.entry(); + let exit = ssa.add_exit(exit_ip, Vec::new(), Vec::new(), Vec::new()); + ssa.set_terminator(entry, SsaTerminator::Exit { exit }) + .unwrap(); + ( + JitTrace { + id, + frame_key: 7, + root_ip, + entry_stack_depth: 0, + start_line: None, + has_call: false, + has_yielding_call: false, + op_names: vec![format!("trace{id}")], + terminal: JitTraceTerminal::BranchExit, + executions: 0, + ssa: ssa.finish(), + }, + exit, + ) + } + + #[test] + fn two_trace_region_remaps_child_ids_and_preserves_exit_identity() { + let (parent, parent_exit) = test_trace(3, 0, 12); + let (child, child_exit) = test_trace(5, 12, 24); + let import = SideTraceImport { + parent_exit, + stack_depth: 0, + local_count: 0, + dirty_locals: Vec::new(), + args: Vec::new(), + }; + + let back_import = SideTraceImport { + parent_exit: child_exit, + stack_depth: 0, + local_count: 0, + dirty_locals: Vec::new(), + args: Vec::new(), + }; + + let region = fuse_two_trace_region(&parent, &child, &import, Some(&back_import)).unwrap(); + + assert_eq!(region.trace.ssa.blocks.len(), 2); + assert_eq!(region.trace.ssa.exits.len(), 2); + assert_eq!(region.links.len(), 2); + assert_eq!(region.links[0].exit, parent_exit); + assert_eq!(region.links[0].child_entry, SsaBlockId::new(1)); + assert!(region.links[0].args.is_empty()); + assert_eq!(region.links[1].exit, SsaExitId::new(1)); + assert_eq!(region.links[1].child_entry, parent.ssa.entry); + assert!(region.links[1].args.is_empty()); + assert_eq!( + region.exit_keys.get(&parent_exit.raw()), + Some(&TraceExitKey { + parent_trace_id: parent.id, + exit_id: parent_exit, + }) + ); + assert_eq!( + region.exit_keys.get(&(child_exit.raw() + 1)), + Some(&TraceExitKey { + parent_trace_id: child.id, + exit_id: child_exit, + }) + ); + } +} diff --git a/src/vm/jit/runtime.rs b/src/vm/jit/runtime.rs index 6bd3b8ca..d8c03970 100644 --- a/src/vm/jit/runtime.rs +++ b/src/vm/jit/runtime.rs @@ -8,6 +8,8 @@ use super::super::{ExecOutcome, Vm, VmError, VmResult}; all(target_arch = "aarch64", any(target_os = "linux", target_os = "macos")) ))] use super::JitTrace; +use super::ir::SsaExitId; +use super::trace::TraceExitKey; use super::{JitMetrics, JitTraceTerminal, native}; use crate::vm::native::ROOT_FRAME_KEY; use std::collections::HashMap; @@ -39,9 +41,37 @@ type NativeTraceEntry = unsafe extern "C" fn(*mut Vm) -> i32; )))] type NativeTraceEntry = fn(*mut Vm) -> i32; +type NativeTraceState = ( + NativeTraceEntry, + usize, + JitTraceTerminal, + bool, + bool, + Option>>, + bool, +); + +fn scalar_cycle_import(import: &super::deopt::SideTraceImport) -> bool { + import.args.iter().all(|arg| { + matches!( + arg, + super::ir::SsaMaterialization::BoxInt(_) + | super::ir::SsaMaterialization::BoxFloat(_) + | super::ir::SsaMaterialization::BoxBool(_) + ) + }) +} + +fn elapsed_ns(started: std::time::Instant) -> u64 { + u64::try_from(started.elapsed().as_nanos()).unwrap_or(u64::MAX) +} + pub(crate) struct NativeTrace { _keepalive: Arc>, + _direct_keepalives: Vec>>, entry: NativeTraceEntry, + tail_entry: NativeTraceEntry, + direct_slots: Arc>>, pub(super) code: Arc<[u8]>, root_ip: usize, terminal: JitTraceTerminal, @@ -51,6 +81,21 @@ pub(crate) struct NativeTrace { interrupt_settings: Option, compile_profile: native::NativeCompileProfile, drop_contract_events_enabled: bool, + region: Option, +} + +struct NativeRegion { + _keepalive: Arc>, + entry: NativeTraceEntry, + code: Arc<[u8]>, + terminal: JitTraceTerminal, + has_call: bool, + has_yielding_call: bool, + lowering_kind: native::TraceLoweringKind, + generation: u64, + key: TraceExitKey, + child_trace_id: usize, + exit_keys: Arc>, } #[cfg(any( @@ -82,6 +127,7 @@ struct NativeTraceCacheKey { #[derive(Clone)] struct NativeTraceCacheEntry { entry: NativeTraceEntry, + tail_entry: NativeTraceEntry, keepalive: Arc>, code: Arc<[u8]>, lowering_kind: native::TraceLoweringKind, @@ -185,6 +231,17 @@ pub(crate) fn resume_linked_trace_entry_address() -> usize { pd_vm_native_resume_linked_trace as *const () as usize } +#[cfg(not(any( + all( + target_arch = "x86_64", + any(target_os = "windows", all(unix, not(target_os = "macos"))) + ), + all(target_arch = "aarch64", any(target_os = "linux", target_os = "macos")) +)))] +pub(crate) fn resume_linked_trace_entry_address() -> usize { + 0 +} + #[cfg(any( all( target_arch = "x86_64", @@ -237,7 +294,7 @@ impl Vm { let mut next_trace_id = self .jit .compiled_trace_for_entry(frame_key, ip, stack_depth); - if next_trace_id.is_none() { + if next_trace_id.is_none() && !self.jit.callable_frame_is_blocked(frame_key) { let entry_local_types = (frame_key != ROOT_FRAME_KEY).then(|| self.active_local_types()); let program = &self.program; @@ -261,20 +318,72 @@ impl Vm { { if should_fallback_to_interpreter(&err) { self.record_jit_helper_fallback(); - self.jit.block_trace(current_trace_id); + self.block_jit_trace(current_trace_id); return Ok(native::STATUS_LINKED_CONTINUE); } return Err(err); } - let (mut entry, mut root_ip, mut terminal, _, mut has_yielding_call) = - self.native_trace_state(current_trace_id)?; + let ( + mut entry, + mut root_ip, + mut terminal, + _, + mut has_yielding_call, + mut exit_keys, + mut is_region, + ) = self.native_trace_state(current_trace_id)?; loop { native::clear_bridge_error(); + let region_edges_before = self.jit_native_region_edge_count; + let direct_links_before = self.jit_native_direct_link_count; let status = unsafe { entry(self as *mut Vm) }; self.native_trace_exec_count = self.native_trace_exec_count.saturating_add(1); + if !is_region + && self.jit_native_active_direct_trace_id != usize::MAX + && self.jit_native_active_direct_trace_id != current_trace_id + { + current_trace_id = self.jit_native_active_direct_trace_id; + let state = self.native_trace_state(current_trace_id)?; + entry = state.0; + root_ip = state.1; + terminal = state.2; + has_yielding_call = state.4; + exit_keys = state.5; + is_region = state.6; + } + self.record_native_direct_escape(status, direct_links_before); + if is_region { + self.jit_native_region_entry_count = + self.jit_native_region_entry_count.saturating_add(1); + if self.jit_native_region_edge_count > region_edges_before { + self.jit.record_native_region_progress(current_trace_id); + } + } self.jit.mark_trace_executed(current_trace_id); + let mut trace_exit_key = None; + let status = if let Some(exit_id) = native::decode_jit_trace_exit_status(status) { + let key = if let Some(region_exit_keys) = &exit_keys { + *region_exit_keys.get(&exit_id).ok_or_else(|| { + VmError::JitNative(format!( + "fused native region returned impossible exit id {exit_id}" + )) + })? + } else { + TraceExitKey { + parent_trace_id: current_trace_id, + exit_id: SsaExitId::new(exit_id), + } + }; + self.jit + .record_trace_exit(key) + .map_err(|err| VmError::JitNative(err.message()))?; + trace_exit_key = Some(key); + native::STATUS_TRACE_EXIT + } else { + status + }; match status { native::STATUS_CONTINUE => { @@ -290,6 +399,11 @@ impl Vm { if let Some(next_trace_id) = next_trace_id && next_trace_id != current_trace_id { + self.publish_native_direct_slot( + current_trace_id, + native::CONTINUE_SLOT_ID, + next_trace_id, + )?; self.record_jit_link_handoff(); current_trace_id = next_trace_id; if let Err(err) = self.ensure_native_trace( @@ -298,13 +412,20 @@ impl Vm { ) { if should_fallback_to_interpreter(&err) { self.record_jit_helper_fallback(); - self.jit.block_trace(current_trace_id); + self.block_jit_trace(current_trace_id); return Ok(native::STATUS_LINKED_CONTINUE); } return Err(err); } - (entry, root_ip, terminal, _, has_yielding_call) = - self.native_trace_state(current_trace_id)?; + ( + entry, + root_ip, + terminal, + _, + has_yielding_call, + exit_keys, + is_region, + ) = self.native_trace_state(current_trace_id)?; continue; } return Ok(native::STATUS_LINKED_CONTINUE); @@ -315,10 +436,17 @@ impl Vm { && terminal == JitTraceTerminal::LoopBack && self.ip == root_ip { + self.jit.record_native_loop_back(current_trace_id); self.jit_native_loop_back_count = self.jit_native_loop_back_count.saturating_add(1); continue; } + if self.jit.record_native_side_exit(current_trace_id) + && !self.jit_native_direct_links_enabled + { + self.block_jit_callable_frame(current_trace_id); + return Ok(native::STATUS_LINKED_CONTINUE); + } if !has_yielding_call { let ip = self.ip; let frame_key = self.active_frame_key(); @@ -326,7 +454,8 @@ impl Vm { let mut next_trace_id = self.jit .compiled_trace_for_entry(frame_key, ip, stack_depth); - if next_trace_id.is_none() { + if next_trace_id.is_none() && !self.jit.callable_frame_is_blocked(frame_key) + { let entry_local_types = (frame_key != ROOT_FRAME_KEY).then(|| self.active_local_types()); let program = &self.program; @@ -341,6 +470,10 @@ impl Vm { if let Some(next_trace_id) = next_trace_id && next_trace_id != current_trace_id { + if let Some(key) = trace_exit_key { + self.publish_native_direct_link(key, next_trace_id)?; + self.maybe_publish_native_region(key, next_trace_id); + } self.record_jit_link_handoff(); current_trace_id = next_trace_id; if let Err(err) = self.ensure_native_trace( @@ -349,13 +482,20 @@ impl Vm { ) { if should_fallback_to_interpreter(&err) { self.record_jit_helper_fallback(); - self.jit.block_trace(current_trace_id); + self.block_jit_trace(current_trace_id); return Ok(native::STATUS_LINKED_CONTINUE); } return Err(err); } - (entry, root_ip, terminal, _, has_yielding_call) = - self.native_trace_state(current_trace_id)?; + ( + entry, + root_ip, + terminal, + _, + has_yielding_call, + exit_keys, + is_region, + ) = self.native_trace_state(current_trace_id)?; continue; } } @@ -388,12 +528,241 @@ impl Vm { } } + fn clear_native_direct_links(&self) { + for native in self.native_traces.iter().flatten() { + for slot in native.direct_slots.values() { + slot.clear(); + } + } + } + + fn record_native_direct_escape(&mut self, _status: i32, direct_links_before: u64) { + if !self.jit_native_direct_links_enabled + || self.jit_native_direct_link_count == direct_links_before + { + return; + } + self.jit_native_direct_escape_streak = 0; + if self.jit_native_active_direct_trace_id != usize::MAX { + self.jit + .record_native_loop_back(self.jit_native_active_direct_trace_id); + } + } + + fn publish_native_direct_link( + &mut self, + key: TraceExitKey, + child_trace_id: usize, + ) -> VmResult<()> { + if !self.jit_native_direct_links_enabled || self.jit_native_direct_region_fallback { + return Ok(()); + } + self.publish_native_direct_slot(key.parent_trace_id, key.exit_id.raw(), child_trace_id) + } + + #[cfg(any( + all( + target_arch = "x86_64", + any(target_os = "windows", all(unix, not(target_os = "macos"))) + ), + all(target_arch = "aarch64", any(target_os = "linux", target_os = "macos")) + ))] + fn publish_native_direct_slot( + &mut self, + parent_trace_id: usize, + slot_id: u32, + child_trace_id: usize, + ) -> VmResult<()> { + if !self.jit_native_direct_cross_frame_enabled { + let parent_frame_key = self + .jit + .trace_clone(parent_trace_id) + .map(|trace| trace.frame_key); + let child_frame_key = self + .jit + .trace_clone(child_trace_id) + .map(|trace| trace.frame_key); + if parent_frame_key != child_frame_key { + return Ok(()); + } + } + self.ensure_native_trace(child_trace_id, native::NativeCompileProfile::Jit)?; + let child_entry = self + .native_traces + .get(child_trace_id) + .and_then(Option::as_ref) + .ok_or_else(|| { + VmError::JitNative("direct-link child native trace missing".to_string()) + })? + .tail_entry as *const u8; + let Some(slot) = self + .native_traces + .get(parent_trace_id) + .and_then(Option::as_ref) + .and_then(|native| native.direct_slots.get(&slot_id)) + .cloned() + else { + return Ok(()); + }; + slot.publish(child_entry); + Ok(()) + } + + #[cfg(not(any( + all( + target_arch = "x86_64", + any(target_os = "windows", all(unix, not(target_os = "macos"))) + ), + all(target_arch = "aarch64", any(target_os = "linux", target_os = "macos")) + )))] + fn publish_native_direct_slot( + &mut self, + _parent_trace_id: usize, + _slot_id: u32, + _child_trace_id: usize, + ) -> VmResult<()> { + Ok(()) + } + + #[cfg(any( + all( + target_arch = "x86_64", + any(target_os = "windows", all(unix, not(target_os = "macos"))) + ), + all(target_arch = "aarch64", any(target_os = "linux", target_os = "macos")) + ))] + fn maybe_publish_native_region(&mut self, key: TraceExitKey, child_trace_id: usize) { + if self.jit_native_direct_links_enabled && !self.jit_native_direct_region_fallback { + return; + } + let Some(candidate) = self.jit.region_candidate(key, child_trace_id) else { + return; + }; + if candidate.generation != self.jit.region_generation() { + return; + } + let Some(parent) = self.jit.trace_clone(key.parent_trace_id) else { + self.jit.record_region_compile_failure(&candidate); + return; + }; + let Some(child) = self.jit.trace_clone(child_trace_id) else { + self.jit.record_region_compile_failure(&candidate); + return; + }; + let back_import = scalar_cycle_import(&candidate.import) + .then(|| { + child + .ssa + .exits + .iter() + .filter(|exit| exit.exit_ip == parent.root_ip) + .find_map(|exit| { + self.jit + .side_trace_import(child.id, exit.id, parent.id) + .ok() + }) + }) + .flatten() + .filter(scalar_cycle_import); + let fused = match super::region::fuse_two_trace_region( + &parent, + &child, + &candidate.import, + back_import.as_ref(), + ) { + Ok(fused) => fused, + Err(_) => { + self.jit.record_region_compile_failure(&candidate); + return; + } + }; + let interrupt_settings = self.active_native_interrupt_settings(); + let compile_profile = native::NativeCompileProfile::Jit; + let drop_contract_events_enabled = self.drop_contract_events_enabled(); + let compile_started = std::time::Instant::now(); + let compile_result = native::compile_native_region( + &fused, + interrupt_settings, + compile_profile, + drop_contract_events_enabled, + ); + self.jit_native_region_compile_time_ns = self + .jit_native_region_compile_time_ns + .saturating_add(elapsed_ns(compile_started)); + let compiled = match compile_result { + Ok(compiled) => compiled, + Err(_) => { + self.jit.record_region_compile_failure(&candidate); + return; + } + }; + let entry = unsafe { std::mem::transmute::<*const u8, NativeTraceEntry>(compiled.entry) }; + let lowering_kind = compiled.lowering_kind; + let code = Arc::<[u8]>::from(compiled.code.into_boxed_slice()); + let keepalive = Arc::new(Mutex::new(compiled.keepalive)); + let region = NativeRegion { + _keepalive: keepalive, + entry, + code, + terminal: fused.trace.terminal, + has_call: fused.trace.has_call, + has_yielding_call: fused.trace.has_yielding_call, + lowering_kind, + generation: candidate.generation, + key, + child_trace_id, + exit_keys: Arc::new(fused.exit_keys), + }; + let Some(parent_native) = self + .native_traces + .get_mut(key.parent_trace_id) + .and_then(Option::as_mut) + else { + self.jit.record_region_compile_failure(&candidate); + return; + }; + if !self.jit.publish_region(&candidate) { + return; + } + parent_native.region = Some(region); + } + + fn clear_native_region_owners(&mut self) { + for native in self.native_traces.iter_mut().flatten() { + native.region = None; + } + } + + pub(crate) fn disconnect_native_regions(&mut self) { + self.jit.invalidate_regions(); + self.clear_native_region_owners(); + } + + fn block_jit_trace(&mut self, trace_id: usize) { + self.jit.block_trace(trace_id); + self.clear_native_region_owners(); + } + + fn block_jit_callable_frame(&mut self, trace_id: usize) { + self.jit.block_callable_frame(trace_id); + self.clear_native_region_owners(); + } + pub fn set_jit_config(&mut self, config: super::JitConfig) { if config.enabled { self.ensure_program_cache_key(); } + self.clear_native_direct_links(); self.native_traces.clear(); self.native_trace_exec_count = 0; + self.jit_native_region_entry_count = 0; + self.jit_native_region_edge_count = 0; + self.jit_native_direct_link_count = 0; + self.jit_native_active_direct_trace_id = usize::MAX; + self.jit_native_direct_escape_streak = 0; + self.jit_native_direct_region_fallback = false; + self.jit_native_compile_time_ns = 0; + self.jit_native_region_compile_time_ns = 0; self.jit_trace_exit_count = 0; self.jit_native_loop_back_count = 0; self.jit_native_link_handoff_count = 0; @@ -410,6 +779,35 @@ impl Vm { self.jit.snapshot(self.jit_runtime_metrics()) } + pub fn jit_exit_profiles(&self) -> Vec { + self.jit.exit_profiles() + } + + pub fn jit_native_code_bytes(&self) -> usize { + self.native_traces + .iter() + .flatten() + .map(|native| native.code.len()) + .sum() + } + + pub fn jit_native_region_code_bytes(&self) -> usize { + self.native_traces + .iter() + .flatten() + .filter_map(|native| native.region.as_ref()) + .map(|region| region.code.len()) + .sum() + } + + pub fn jit_native_compile_time_ns(&self) -> u64 { + self.jit_native_compile_time_ns + } + + pub fn jit_native_region_compile_time_ns(&self) -> u64 { + self.jit_native_region_compile_time_ns + } + pub fn dump_jit_info(&self) -> String { self.dump_jit_info_with_machine_code(true) } @@ -430,6 +828,27 @@ impl Vm { " native trace handoffs: {}\n", self.jit_native_link_handoff_count )); + out.push_str(&format!( + " native region entries: {}\n", + self.jit_native_region_entry_count + )); + out.push_str(&format!( + " native internal region edges: {}\n", + self.jit_native_region_edge_count + )); + out.push_str(&format!( + " native direct side links: {}\n", + self.jit_native_direct_link_count + )); + out.push_str(&format!( + " native compile time: {} ns (regions={} ns)\n", + self.jit_native_compile_time_ns, self.jit_native_region_compile_time_ns + )); + out.push_str(&format!( + " native code bytes: {} (regions={})\n", + self.jit_native_code_bytes(), + self.jit_native_region_code_bytes() + )); if self.jit_native_bridge_stats_enabled { let mut bridge_entries: Vec<(&'static str, u64)> = self .jit_native_bridge_counts @@ -472,6 +891,21 @@ impl Vm { } out.push('\n'); } + if let Some(region) = &native.region { + out.push_str(&format!( + " region entry=0x{:X} code_bytes={} lowering={}\n", + region.entry as usize, + region.code.len(), + region.lowering_kind.as_str() + )); + if include_machine_code { + out.push_str(" code:"); + for byte in region.code.iter() { + out.push_str(&format!(" {:02X}", byte)); + } + out.push('\n'); + } + } } } out @@ -490,7 +924,7 @@ impl Vm { Ok(outcome) => Ok(outcome), Err(err) if should_fallback_to_interpreter(&err) => { self.record_jit_helper_fallback(); - self.jit.block_trace(trace_id); + self.block_jit_trace(trace_id); Ok(ExecOutcome::Continue) } Err(err) => Err(err), @@ -523,18 +957,70 @@ impl Vm { { if should_fallback_to_interpreter(&err) { self.record_jit_helper_fallback(); - self.jit.block_trace(current_trace_id); + self.block_jit_trace(current_trace_id); return Ok(ExecOutcome::Continue); } return Err(err); } - let (mut entry, mut root_ip, mut terminal, _, mut has_yielding_call) = - self.native_trace_state(current_trace_id)?; + let ( + mut entry, + mut root_ip, + mut terminal, + _, + mut has_yielding_call, + mut exit_keys, + mut is_region, + ) = self.native_trace_state(current_trace_id)?; native::clear_bridge_error(); loop { + let region_edges_before = self.jit_native_region_edge_count; + let direct_links_before = self.jit_native_direct_link_count; let status = unsafe { entry(self as *mut Vm) }; self.native_trace_exec_count = self.native_trace_exec_count.saturating_add(1); + if !is_region + && self.jit_native_active_direct_trace_id != usize::MAX + && self.jit_native_active_direct_trace_id != current_trace_id + { + current_trace_id = self.jit_native_active_direct_trace_id; + let state = self.native_trace_state(current_trace_id)?; + entry = state.0; + root_ip = state.1; + terminal = state.2; + has_yielding_call = state.4; + exit_keys = state.5; + is_region = state.6; + } + self.record_native_direct_escape(status, direct_links_before); + if is_region { + self.jit_native_region_entry_count = + self.jit_native_region_entry_count.saturating_add(1); + if self.jit_native_region_edge_count > region_edges_before { + self.jit.record_native_region_progress(current_trace_id); + } + } self.jit.mark_trace_executed(current_trace_id); + let mut trace_exit_key = None; + let status = if let Some(exit_id) = native::decode_jit_trace_exit_status(status) { + let key = if let Some(region_exit_keys) = &exit_keys { + *region_exit_keys.get(&exit_id).ok_or_else(|| { + VmError::JitNative(format!( + "fused native region returned impossible exit id {exit_id}" + )) + })? + } else { + TraceExitKey { + parent_trace_id: current_trace_id, + exit_id: SsaExitId::new(exit_id), + } + }; + self.jit + .record_trace_exit(key) + .map_err(|err| VmError::JitNative(err.message()))?; + trace_exit_key = Some(key); + native::STATUS_TRACE_EXIT + } else { + status + }; match status { native::STATUS_CONTINUE => { @@ -550,13 +1036,26 @@ impl Vm { if let Some(next_trace_id) = next_trace_id && next_trace_id != current_trace_id { + self.publish_native_direct_slot( + current_trace_id, + native::CONTINUE_SLOT_ID, + next_trace_id, + )?; self.record_jit_link_handoff(); current_trace_id = next_trace_id; if let Some(state) = self.cached_native_trace_state( current_trace_id, native::NativeCompileProfile::Jit, ) { - (entry, root_ip, terminal, _, has_yielding_call) = state; + ( + entry, + root_ip, + terminal, + _, + has_yielding_call, + exit_keys, + is_region, + ) = state; } else { if let Err(err) = self.ensure_native_trace( current_trace_id, @@ -564,13 +1063,20 @@ impl Vm { ) { if should_fallback_to_interpreter(&err) { self.record_jit_helper_fallback(); - self.jit.block_trace(current_trace_id); + self.block_jit_trace(current_trace_id); return Ok(ExecOutcome::Continue); } return Err(err); } - (entry, root_ip, terminal, _, has_yielding_call) = - self.native_trace_state(current_trace_id)?; + ( + entry, + root_ip, + terminal, + _, + has_yielding_call, + exit_keys, + is_region, + ) = self.native_trace_state(current_trace_id)?; } continue; } @@ -581,7 +1087,7 @@ impl Vm { if self.jit.trace_clone(current_trace_id).is_some_and(|trace| { trace.op_names.last().map(String::as_str) == Some("callable_boundary") }) { - self.jit.block_trace(current_trace_id); + self.block_jit_trace(current_trace_id); return Ok(ExecOutcome::Continue); } // Fast path: if this trace looped back to its own root and cannot yield via host @@ -590,10 +1096,17 @@ impl Vm { && terminal == JitTraceTerminal::LoopBack && self.ip == root_ip { + self.jit.record_native_loop_back(current_trace_id); self.jit_native_loop_back_count = self.jit_native_loop_back_count.saturating_add(1); continue; } + if self.jit.record_native_side_exit(current_trace_id) + && !self.jit_native_direct_links_enabled + { + self.block_jit_callable_frame(current_trace_id); + return Ok(ExecOutcome::Continue); + } if !has_yielding_call { let ip = self.ip; let frame_key = self.active_frame_key(); @@ -601,7 +1114,8 @@ impl Vm { let mut next_trace_id = self.jit .compiled_trace_for_entry(frame_key, ip, stack_depth); - if next_trace_id.is_none() { + if next_trace_id.is_none() && !self.jit.callable_frame_is_blocked(frame_key) + { next_trace_id = { let entry_local_types = (frame_key != ROOT_FRAME_KEY) .then(|| self.active_local_types()); @@ -618,13 +1132,25 @@ impl Vm { if let Some(next_trace_id) = next_trace_id && next_trace_id != current_trace_id { + if let Some(key) = trace_exit_key { + self.publish_native_direct_link(key, next_trace_id)?; + self.maybe_publish_native_region(key, next_trace_id); + } self.record_jit_link_handoff(); current_trace_id = next_trace_id; if let Some(state) = self.cached_native_trace_state( current_trace_id, native::NativeCompileProfile::Jit, ) { - (entry, root_ip, terminal, _, has_yielding_call) = state; + ( + entry, + root_ip, + terminal, + _, + has_yielding_call, + exit_keys, + is_region, + ) = state; } else { if let Err(err) = self.ensure_native_trace( current_trace_id, @@ -632,13 +1158,20 @@ impl Vm { ) { if should_fallback_to_interpreter(&err) { self.record_jit_helper_fallback(); - self.jit.block_trace(current_trace_id); + self.block_jit_trace(current_trace_id); return Ok(ExecOutcome::Continue); } return Err(err); } - (entry, root_ip, terminal, _, has_yielding_call) = - self.native_trace_state(current_trace_id)?; + ( + entry, + root_ip, + terminal, + _, + has_yielding_call, + exit_keys, + is_region, + ) = self.native_trace_state(current_trace_id)?; } continue; } @@ -653,7 +1186,7 @@ impl Vm { let mut next_trace_id = self.jit .compiled_trace_for_entry(frame_key, ip, stack_depth); - if next_trace_id.is_none() { + if next_trace_id.is_none() && !self.jit.callable_frame_is_blocked(frame_key) { next_trace_id = { let entry_local_types = (frame_key != ROOT_FRAME_KEY).then(|| self.active_local_types()); @@ -676,7 +1209,15 @@ impl Vm { current_trace_id, native::NativeCompileProfile::Jit, ) { - (entry, root_ip, terminal, _, has_yielding_call) = state; + ( + entry, + root_ip, + terminal, + _, + has_yielding_call, + exit_keys, + is_region, + ) = state; } else { if let Err(err) = self.ensure_native_trace( current_trace_id, @@ -684,13 +1225,20 @@ impl Vm { ) { if should_fallback_to_interpreter(&err) { self.record_jit_helper_fallback(); - self.jit.block_trace(current_trace_id); + self.block_jit_trace(current_trace_id); return Ok(ExecOutcome::Continue); } return Err(err); } - (entry, root_ip, terminal, _, has_yielding_call) = - self.native_trace_state(current_trace_id)?; + ( + entry, + root_ip, + terminal, + _, + has_yielding_call, + exit_keys, + is_region, + ) = self.native_trace_state(current_trace_id)?; } continue; } @@ -761,10 +1309,7 @@ impl Vm { ), all(target_arch = "aarch64", any(target_os = "linux", target_os = "macos")) ))] - fn native_trace_state( - &self, - trace_id: usize, - ) -> VmResult<(NativeTraceEntry, usize, JitTraceTerminal, bool, bool)> { + fn native_trace_state(&self, trace_id: usize) -> VmResult { let native = self .native_traces .get(trace_id) @@ -772,12 +1317,32 @@ impl Vm { .ok_or_else(|| { VmError::JitNative(format!("native trace entry for id {} missing", trace_id)) })?; + if let Some(region) = native.region.as_ref().filter(|region| { + self.jit.published_region().is_some_and(|published| { + published.generation == region.generation + && published.key == region.key + && published.child_trace_id == region.child_trace_id + && published.key.parent_trace_id == trace_id + }) + }) { + return Ok(( + region.entry, + native.root_ip, + region.terminal, + region.has_call, + region.has_yielding_call, + Some(Arc::clone(®ion.exit_keys)), + true, + )); + } Ok(( native.entry, native.root_ip, native.terminal, native.has_call, native.has_yielding_call, + None, + false, )) } @@ -793,18 +1358,13 @@ impl Vm { &self, trace_id: usize, compile_profile: native::NativeCompileProfile, - ) -> Option<(NativeTraceEntry, usize, JitTraceTerminal, bool, bool)> { + ) -> Option { let native = self.native_traces.get(trace_id)?.as_ref()?; (native.interrupt_settings == self.active_native_interrupt_settings() && compile_profile_satisfies(native.compile_profile, compile_profile) && native.drop_contract_events_enabled == self.drop_contract_events_enabled) - .then_some(( - native.entry, - native.root_ip, - native.terminal, - native.has_call, - native.has_yielding_call, - )) + .then(|| self.native_trace_state(trace_id).ok()) + .flatten() } #[cfg(any( @@ -843,6 +1403,15 @@ impl Vm { { return Ok(()); } + if self + .native_traces + .get(trace_id) + .and_then(Option::as_ref) + .is_some_and(|native| native.region.is_some()) + { + self.disconnect_native_regions(); + } + self.clear_native_direct_links(); if let Some(slot) = self.native_traces.get_mut(trace_id) { *slot = None; } @@ -863,19 +1432,36 @@ impl Vm { cache.entries.clear(); cache.active_program_key = Some(program_cache_key); } - if let Some(cached) = cache.entries.get(&key).cloned() { - return Some(cached); - } - None + cache.entries.get(&key).cloned() }); if let Some(cached) = cached { + let dispatcher = native::compile_native_trace_dispatcher( + trace_id, + cached.tail_entry as *const u8, + &trace, + )?; + let entry = + unsafe { std::mem::transmute::<*const u8, NativeTraceEntry>(dispatcher.entry) }; + let tail_entry = unsafe { + std::mem::transmute::<*const u8, NativeTraceEntry>(dispatcher.tail_entry) + }; + let direct_keepalives = dispatcher + .keepalives + .into_iter() + .map(|keepalive| Arc::new(Mutex::new(keepalive))) + .collect(); + let mut code = cached.code.to_vec(); + code.extend_from_slice(&dispatcher.code); if self.native_traces.len() <= trace_id { self.native_traces.resize_with(trace_id + 1, || None); } self.native_traces[trace_id] = Some(NativeTrace { _keepalive: cached.keepalive, - entry: cached.entry, - code: cached.code, + _direct_keepalives: direct_keepalives, + entry, + tail_entry, + direct_slots: Arc::new(dispatcher.slots), + code: Arc::from(code.into_boxed_slice()), root_ip: trace.root_ip, terminal: trace.terminal, has_call: trace.has_call, @@ -884,24 +1470,35 @@ impl Vm { interrupt_settings, compile_profile: cached.compile_profile, drop_contract_events_enabled, + region: None, }); return Ok(()); } - let compiled = native::compile_native_trace( + let compile_started = std::time::Instant::now(); + let compile_result = native::compile_native_trace( &trace, interrupt_settings, compile_profile, drop_contract_events_enabled, - )?; - let entry = unsafe { std::mem::transmute::<*const u8, NativeTraceEntry>(compiled.entry) }; - let code = Arc::<[u8]>::from(compiled.code.into_boxed_slice()); + ); + self.jit_native_compile_time_ns = self + .jit_native_compile_time_ns + .saturating_add(elapsed_ns(compile_started)); + let compiled = compile_result?; + let lowering_kind = compiled.lowering_kind; + let base_entry = + unsafe { std::mem::transmute::<*const u8, NativeTraceEntry>(compiled.entry) }; + let base_tail_entry = + unsafe { std::mem::transmute::<*const u8, NativeTraceEntry>(compiled.tail_entry) }; let keepalive = Arc::new(Mutex::new(compiled.keepalive)); + let base_code = Arc::<[u8]>::from(compiled.code.clone().into_boxed_slice()); let cached = NativeTraceCacheEntry { - entry, + entry: base_entry, + tail_entry: base_tail_entry, keepalive: Arc::clone(&keepalive), - code: Arc::clone(&code), - lowering_kind: compiled.lowering_kind, + code: Arc::clone(&base_code), + lowering_kind, compile_profile, }; with_native_trace_cache(|cache| { @@ -911,21 +1508,39 @@ impl Vm { } cache.entries.insert(key, cached); }); + let dispatcher = + native::compile_native_trace_dispatcher(trace_id, compiled.tail_entry, &trace)?; + let entry = unsafe { std::mem::transmute::<*const u8, NativeTraceEntry>(dispatcher.entry) }; + let tail_entry = + unsafe { std::mem::transmute::<*const u8, NativeTraceEntry>(dispatcher.tail_entry) }; + let direct_keepalives = dispatcher + .keepalives + .into_iter() + .map(|keepalive| Arc::new(Mutex::new(keepalive))) + .collect(); + let direct_slots = Arc::new(dispatcher.slots); + let mut code = compiled.code; + code.extend_from_slice(&dispatcher.code); + let code = Arc::<[u8]>::from(code.into_boxed_slice()); if self.native_traces.len() <= trace_id { self.native_traces.resize_with(trace_id + 1, || None); } self.native_traces[trace_id] = Some(NativeTrace { _keepalive: keepalive, + _direct_keepalives: direct_keepalives, entry, + tail_entry, + direct_slots, code, root_ip: trace.root_ip, terminal: trace.terminal, has_call: trace.has_call, has_yielding_call: trace.has_yielding_call, - lowering_kind: compiled.lowering_kind, + lowering_kind, interrupt_settings, compile_profile, drop_contract_events_enabled, + region: None, }); Ok(()) } @@ -938,6 +1553,75 @@ impl Vm { self.native_trace_exec_count } + pub(crate) fn jit_native_inherited_target(&self) -> usize { + if !self.jit_native_direct_links_enabled { + return 0; + } + let Some(trace_id) = self.jit.compiled_trace_for_entry( + self.active_frame_key(), + self.ip, + self.active_operand_stack_len(), + ) else { + return 0; + }; + self.native_traces + .get(trace_id) + .and_then(Option::as_ref) + .map(|native| native.tail_entry as usize) + .unwrap_or(0) + } + + pub fn set_jit_native_direct_links_enabled(&mut self, enabled: bool) { + let cross_frame_enabled = enabled; + if self.jit_native_direct_links_enabled == enabled + && self.jit_native_direct_cross_frame_enabled == cross_frame_enabled + { + return; + } + self.clear_native_direct_links(); + self.disconnect_native_regions(); + self.native_traces.clear(); + self.jit_native_direct_links_enabled = enabled; + self.jit_native_direct_cross_frame_enabled = cross_frame_enabled; + self.jit_native_direct_link_count = 0; + self.jit_native_active_direct_trace_id = usize::MAX; + self.jit_native_direct_escape_streak = 0; + self.jit_native_direct_region_fallback = false; + } + + pub fn jit_native_region_count(&self) -> usize { + self.native_traces + .iter() + .flatten() + .filter(|native| native.region.is_some()) + .count() + } + + pub fn jit_native_region_entry_count(&self) -> u64 { + self.jit_native_region_entry_count + } + + pub fn jit_native_internal_region_edge_count(&self) -> u64 { + self.jit_native_region_edge_count + } + + pub fn jit_native_direct_link_count(&self) -> u64 { + self.jit_native_direct_link_count + } + + pub fn jit_native_active_direct_link_slot_count(&self) -> usize { + self.native_traces + .iter() + .flatten() + .flat_map(|native| native.direct_slots.values()) + .filter(|slot| !slot.target().is_null()) + .count() + } + + pub fn jit_helper_fallback_count(&self) -> u64 { + self.jit_helper_fallback_count + } + pub fn jit_native_link_handoff_count(&self) -> u64 { self.jit_native_link_handoff_count } @@ -992,3 +1676,36 @@ pub(crate) fn clear_native_trace_cache_for_tests() { pub(crate) fn native_trace_cache_snapshot_for_tests() -> (Option, usize) { with_native_trace_cache(|cache| (cache.active_program_key, cache.entries.len())) } + +#[cfg(test)] +mod tests { + use super::*; + use crate::vm::jit::deopt::SideTraceImport; + use crate::vm::jit::ir::{SsaMaterialization, SsaValueId}; + + fn import_with(arg: SsaMaterialization) -> SideTraceImport { + SideTraceImport { + parent_exit: SsaExitId::new(0), + stack_depth: 1, + local_count: 0, + dirty_locals: Vec::new(), + args: vec![arg], + } + } + + #[test] + fn scalar_cycle_import_rejects_tagged_and_heap_materializations() { + assert!(scalar_cycle_import(&import_with( + SsaMaterialization::BoxInt(SsaValueId::new(0)) + ))); + assert!(!scalar_cycle_import(&import_with( + SsaMaterialization::Value(SsaValueId::new(0)) + ))); + assert!(!scalar_cycle_import(&import_with( + SsaMaterialization::BoxHeapPtr { + value: SsaValueId::new(0), + tag: crate::ValueType::Array, + } + ))); + } +} diff --git a/src/vm/jit/trace.rs b/src/vm/jit/trace.rs index 334e301f..31b02902 100644 --- a/src/vm/jit/trace.rs +++ b/src/vm/jit/trace.rs @@ -4,7 +4,8 @@ use crate::debug_info::DebugInfo; use crate::vm::native::ROOT_FRAME_KEY; use crate::vm::{OpCode, Program}; -use super::ir::SsaTrace; +use super::deopt::{SideTraceImport, SideTraceImportError}; +use super::ir::{SsaExitId, SsaMaterialization, SsaTrace, SsaValueId, SsaValueRepr}; use super::liveness::{boxed_load_site_count, boxed_store_site_count}; use super::recorder::{RecordedTrace, TraceRecordError, record_trace_with_local_count}; @@ -15,6 +16,84 @@ pub(crate) struct TraceEntryKey { pub(crate) stack_depth: usize, } +#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] +pub(crate) struct TraceExitKey { + pub(crate) parent_trace_id: usize, + pub(crate) exit_id: SsaExitId, +} + +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct RegionCandidate { + pub(crate) generation: u64, + pub(crate) key: TraceExitKey, + pub(crate) child_trace_id: usize, + pub(crate) import: SideTraceImport, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) struct PublishedRegion { + pub(crate) generation: u64, + pub(crate) key: TraceExitKey, + pub(crate) child_trace_id: usize, +} + +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub(crate) struct TraceExitProfile { + pub(crate) executions: u64, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub(crate) enum TraceExitRecordError { + UnknownParentTrace(usize), + UnknownExit { + parent_trace_id: usize, + exit_id: SsaExitId, + exit_count: usize, + }, +} + +impl TraceExitRecordError { + pub(crate) fn message(self) -> String { + match self { + Self::UnknownParentTrace(parent_trace_id) => { + format!("native trace exit parent trace {parent_trace_id} does not exist") + } + Self::UnknownExit { + parent_trace_id, + exit_id, + exit_count, + } => format!( + "native trace {parent_trace_id} returned impossible exit id {} (parent has {exit_count} exits)", + exit_id.raw() + ), + } + } +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +#[allow(dead_code)] +pub(crate) enum SideTraceAdmissionError { + UnknownParentTrace(usize), + UnknownChildTrace(usize), + SameTrace(usize), + CrossFrame { + parent_frame_key: u64, + child_frame_key: u64, + }, + YieldingTrace(usize), + Import(SideTraceImportError), + UnknownParentValue(SsaValueId), + ReprMismatch { + index: usize, + parent: SsaValueRepr, + child: SsaValueRepr, + }, + UnsupportedHeapOwnership { + index: usize, + value: SsaValueId, + }, +} + #[derive(Clone, Copy, Debug, PartialEq, Eq)] pub struct JitConfig { pub enabled: bool, @@ -39,6 +118,23 @@ fn native_jit_supported() -> bool { && (cfg!(target_os = "linux") || cfg!(target_os = "macos"))) } +#[allow(dead_code)] +fn ssa_value_repr(trace: &SsaTrace, value: SsaValueId) -> Option { + trace.blocks.iter().find_map(|block| { + block + .params + .iter() + .find_map(|param| (param.value.id == value).then_some(param.value.repr)) + .or_else(|| { + block.insts.iter().find_map(|inst| { + inst.output + .filter(|output| output.id == value) + .map(|output| output.repr) + }) + }) + }) +} + #[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)] pub enum JitTraceTerminal { LoopBack, @@ -165,6 +261,13 @@ impl JitMetrics { } } +#[derive(Clone, Debug, PartialEq, Eq)] +pub struct JitExitProfile { + pub parent_trace_id: usize, + pub exit_id: u32, + pub executions: u64, +} + #[derive(Clone, Debug, PartialEq)] pub struct JitSnapshot { pub arch: &'static str, @@ -181,6 +284,11 @@ pub struct JitNyiDoc { pub reason: &'static str, } +const CALLABLE_SIDE_EXIT_BACKOFF_THRESHOLD: u32 = 64; +const REGION_EXIT_HOT_THRESHOLD: u64 = 2; +const MAX_REGION_MEMBERS: usize = 2; +const MAX_REGION_LINKS: usize = 2; + pub struct TraceJitEngine { config: JitConfig, hot_counts: HashMap, @@ -189,6 +297,12 @@ pub struct TraceJitEngine { loop_headers: Option>, non_yielding_host_imports: Vec, traces: Vec, + trace_exit_profiles: HashMap, + region_generation: u64, + published_region: Option, + failed_region_exits: HashSet, + callable_side_exit_streaks: Vec, + blocked_callable_frames: Vec, attempts: Vec, } @@ -208,6 +322,12 @@ impl TraceJitEngine { loop_headers: None, non_yielding_host_imports: Vec::new(), traces: Vec::new(), + trace_exit_profiles: HashMap::new(), + region_generation: 0, + published_region: None, + failed_region_exits: HashSet::new(), + callable_side_exit_streaks: Vec::new(), + blocked_callable_frames: Vec::new(), attempts: Vec::new(), } } @@ -217,12 +337,16 @@ impl TraceJitEngine { } pub fn set_config(&mut self, config: JitConfig) { + self.invalidate_regions(); self.config = config; self.hot_counts.clear(); self.compiled_by_ip.clear(); self.blocked_entries.clear(); self.loop_headers = None; self.traces.clear(); + self.trace_exit_profiles.clear(); + self.callable_side_exit_streaks.clear(); + self.blocked_callable_frames.clear(); self.attempts.clear(); } @@ -230,12 +354,16 @@ impl TraceJitEngine { if self.non_yielding_host_imports == imports { return false; } + self.invalidate_regions(); self.non_yielding_host_imports = imports; self.hot_counts.clear(); self.compiled_by_ip.clear(); self.blocked_entries.clear(); self.loop_headers = None; self.traces.clear(); + self.trace_exit_profiles.clear(); + self.callable_side_exit_streaks.clear(); + self.blocked_callable_frames.clear(); self.attempts.clear(); true } @@ -262,7 +390,10 @@ impl TraceJitEngine { entry_local_types: Option<&[crate::ValueType]>, program: &Program, ) -> Option { - if !self.config.enabled || !native_jit_supported() { + if !self.config.enabled + || !native_jit_supported() + || self.callable_frame_is_blocked(frame_key) + { return None; } let key = TraceEntryKey { @@ -301,6 +432,153 @@ impl TraceJitEngine { self.traces.get(trace_id).cloned() } + pub(crate) fn region_generation(&self) -> u64 { + self.region_generation + } + + fn region_exit_is_hot(&self, key: TraceExitKey) -> bool { + self.trace_exit_profiles + .get(&key) + .is_some_and(|profile| profile.executions >= REGION_EXIT_HOT_THRESHOLD) + } + + pub(crate) fn region_candidate( + &mut self, + key: TraceExitKey, + child_trace_id: usize, + ) -> Option { + if MAX_REGION_MEMBERS != 2 + || MAX_REGION_LINKS != 2 + || self.published_region.is_some() + || self.failed_region_exits.contains(&key) + || !self.region_exit_is_hot(key) + { + return None; + } + let parent = self.traces.get(key.parent_trace_id)?; + let child = self.traces.get(child_trace_id)?; + + let region_ops = parent.op_names.len().checked_add(child.op_names.len()); + if region_ops.is_none_or(|ops| ops > self.config.max_trace_len) { + self.failed_region_exits.insert(key); + return None; + } + let import = match self.side_trace_import(key.parent_trace_id, key.exit_id, child_trace_id) + { + Ok(import) => import, + Err(_) => { + self.failed_region_exits.insert(key); + return None; + } + }; + Some(RegionCandidate { + generation: self.region_generation, + key, + child_trace_id, + import, + }) + } + + pub(crate) fn publish_region(&mut self, candidate: &RegionCandidate) -> bool { + if candidate.generation != self.region_generation || self.published_region.is_some() { + return false; + } + let Ok(import) = self.side_trace_import( + candidate.key.parent_trace_id, + candidate.key.exit_id, + candidate.child_trace_id, + ) else { + return false; + }; + if import != candidate.import { + return false; + } + self.published_region = Some(PublishedRegion { + generation: candidate.generation, + key: candidate.key, + child_trace_id: candidate.child_trace_id, + }); + true + } + + pub(crate) fn record_region_compile_failure(&mut self, candidate: &RegionCandidate) { + if candidate.generation == self.region_generation { + self.failed_region_exits.insert(candidate.key); + } + } + + pub(crate) fn published_region(&self) -> Option { + self.published_region + } + + pub(crate) fn invalidate_regions(&mut self) { + self.region_generation = self.region_generation.wrapping_add(1); + self.published_region = None; + self.failed_region_exits.clear(); + } + + pub(crate) fn side_trace_import( + &self, + parent_trace_id: usize, + parent_exit_id: SsaExitId, + child_trace_id: usize, + ) -> Result { + let parent = self + .traces + .get(parent_trace_id) + .ok_or(SideTraceAdmissionError::UnknownParentTrace(parent_trace_id))?; + let child = self + .traces + .get(child_trace_id) + .ok_or(SideTraceAdmissionError::UnknownChildTrace(child_trace_id))?; + if parent_trace_id == child_trace_id { + return Err(SideTraceAdmissionError::SameTrace(parent_trace_id)); + } + if parent.frame_key != child.frame_key { + return Err(SideTraceAdmissionError::CrossFrame { + parent_frame_key: parent.frame_key, + child_frame_key: child.frame_key, + }); + } + if parent.has_yielding_call { + return Err(SideTraceAdmissionError::YieldingTrace(parent_trace_id)); + } + if child.has_yielding_call { + return Err(SideTraceAdmissionError::YieldingTrace(child_trace_id)); + } + let import = super::deopt::side_trace_import(&parent.ssa, parent_exit_id, &child.ssa) + .map_err(SideTraceAdmissionError::Import)?; + let child_entry = child.ssa.blocks.get(child.ssa.entry.index()).ok_or( + SideTraceAdmissionError::Import(SideTraceImportError::InvalidChildEntry), + )?; + for (index, (materialization, child_param)) in + import.args.iter().zip(&child_entry.params).enumerate() + { + let parent_repr = match materialization { + SsaMaterialization::Value(value) => ssa_value_repr(&parent.ssa, *value) + .ok_or(SideTraceAdmissionError::UnknownParentValue(*value))?, + SsaMaterialization::BoxInt(_) + | SsaMaterialization::BoxFloat(_) + | SsaMaterialization::BoxBool(_) => SsaValueRepr::Tagged, + SsaMaterialization::BoxHeapPtr { value, .. } => { + return Err(SideTraceAdmissionError::UnsupportedHeapOwnership { + index, + value: *value, + }); + } + }; + let child_repr = child_param.value.repr; + if parent_repr != child_repr { + return Err(SideTraceAdmissionError::ReprMismatch { + index, + parent: parent_repr, + child: child_repr, + }); + } + } + Ok(import) + } + pub fn observe_exit_ip(&mut self, ip: usize, program: &Program) -> Option { self.observe_exit_entry(ROOT_FRAME_KEY, ip, 0, program) } @@ -323,7 +601,10 @@ impl TraceJitEngine { entry_local_types: Option<&[crate::ValueType]>, program: &Program, ) -> Option { - if !self.config.enabled || !native_jit_supported() { + if !self.config.enabled + || !native_jit_supported() + || self.callable_frame_is_blocked(frame_key) + { return None; } let key = TraceEntryKey { @@ -385,7 +666,117 @@ impl TraceJitEngine { } } + pub(crate) fn record_trace_exit( + &mut self, + key: TraceExitKey, + ) -> Result<(), TraceExitRecordError> { + let trace = self.traces.get(key.parent_trace_id).ok_or( + TraceExitRecordError::UnknownParentTrace(key.parent_trace_id), + )?; + let exit_count = trace.ssa.exits.len(); + if !trace.ssa.exits.iter().any(|exit| exit.id == key.exit_id) { + return Err(TraceExitRecordError::UnknownExit { + parent_trace_id: key.parent_trace_id, + exit_id: key.exit_id, + exit_count, + }); + } + let profile = self.trace_exit_profiles.entry(key).or_default(); + profile.executions = profile.executions.saturating_add(1); + Ok(()) + } + + #[cfg(test)] + pub(crate) fn trace_exit_profile(&self, key: TraceExitKey) -> Option<&TraceExitProfile> { + self.trace_exit_profiles.get(&key) + } + + pub(crate) fn record_native_side_exit(&mut self, trace_id: usize) -> bool { + if self + .traces + .get(trace_id) + .is_none_or(|trace| trace.frame_key == ROOT_FRAME_KEY) + { + return false; + } + let Some(streak) = self.callable_side_exit_streaks.get_mut(trace_id) else { + return false; + }; + *streak = streak.saturating_add(1); + *streak >= CALLABLE_SIDE_EXIT_BACKOFF_THRESHOLD + } + + pub(crate) fn record_native_loop_back(&mut self, trace_id: usize) { + if let Some(streak) = self.callable_side_exit_streaks.get_mut(trace_id) { + *streak = 0; + } + } + + pub(crate) fn record_native_region_progress(&mut self, trace_id: usize) { + self.record_native_loop_back(trace_id); + } + + pub(crate) fn reset_runtime_backoff(&mut self) { + self.callable_side_exit_streaks.fill(0); + self.blocked_callable_frames.fill(false); + self.compiled_by_ip.clear(); + let entries = self + .traces + .iter() + .enumerate() + .map(|(trace_id, trace)| { + ( + TraceEntryKey { + frame_key: trace.frame_key, + root_ip: trace.root_ip, + stack_depth: trace.entry_stack_depth, + }, + trace_id, + ) + }) + .filter(|(key, _)| !self.blocked_entries.contains(key)) + .collect::>(); + for (key, trace_id) in entries { + self.insert_compiled_trace(key, trace_id); + } + } + + pub(crate) fn callable_frame_is_blocked(&self, frame_key: u64) -> bool { + if frame_key == ROOT_FRAME_KEY { + return false; + } + usize::try_from(frame_key) + .ok() + .and_then(|index| self.blocked_callable_frames.get(index)) + .copied() + .unwrap_or(false) + } + + pub(crate) fn block_callable_frame(&mut self, trace_id: usize) { + let Some(frame_key) = self.traces.get(trace_id).map(|trace| trace.frame_key) else { + return; + }; + if frame_key == ROOT_FRAME_KEY { + self.block_trace(trace_id); + return; + } + self.invalidate_regions(); + let Ok(frame_index) = usize::try_from(frame_key) else { + self.block_trace(trace_id); + return; + }; + if self.blocked_callable_frames.len() <= frame_index { + self.blocked_callable_frames.resize(frame_index + 1, false); + } + self.blocked_callable_frames[frame_index] = true; + for entries in &mut self.compiled_by_ip { + entries.retain(|(entry_frame_key, _, _)| *entry_frame_key != frame_key); + } + self.hot_counts.retain(|key, _| key.frame_key != frame_key); + } + pub(crate) fn block_trace(&mut self, trace_id: usize) { + self.invalidate_regions(); if let Some(trace) = self.traces.get(trace_id) { let key = TraceEntryKey { frame_key: trace.frame_key, @@ -397,6 +788,20 @@ impl TraceJitEngine { } } + pub fn exit_profiles(&self) -> Vec { + let mut profiles = self + .trace_exit_profiles + .iter() + .map(|(key, profile)| JitExitProfile { + parent_trace_id: key.parent_trace_id, + exit_id: key.exit_id.raw(), + executions: profile.executions, + }) + .collect::>(); + profiles.sort_by_key(|profile| (profile.parent_trace_id, profile.exit_id)); + profiles + } + pub fn snapshot(&self, runtime_metrics: JitMetrics) -> JitSnapshot { JitSnapshot { arch: std::env::consts::ARCH, @@ -420,6 +825,10 @@ impl TraceJitEngine { )); out.push_str(&format!(" max_trace_len: {}\n", self.config.max_trace_len)); out.push_str(&format!(" compiled traces: {}\n", self.traces.len())); + out.push_str(&format!( + " profiled trace exits: {}\n", + self.trace_exit_profiles.len() + )); out.push_str(&format!(" compile attempts: {}\n", self.attempts.len())); out.push_str(&format!( " boxed value sites: loads={} stores={}\n", @@ -570,6 +979,7 @@ impl TraceJitEngine { .and_then(|debug| debug.line_for_offset(key.root_ip)); let trace = build_jit_trace(id, key, start_line, recorded); self.traces.push(trace); + self.callable_side_exit_streaks.push(0); Ok(id) } @@ -761,10 +1171,458 @@ fn nyi_reference() -> Vec { #[cfg(test)] mod tests { use super::*; + use crate::vm::jit::ir::{ + SsaExitId, SsaMaterialization, SsaTerminator, SsaTrace, SsaTraceBuilder, SsaValueRepr, + }; use crate::{ BytecodeBuilder, CallableKind, CallablePrototype, CallableTarget, ScriptFunction, Value, + ValueType, }; + fn test_trace(id: usize, frame_key: u64, has_yielding_call: bool, ssa: SsaTrace) -> JitTrace { + JitTrace { + id, + frame_key, + root_ip: ssa.root_ip, + entry_stack_depth: ssa.entry_stack_depth, + start_line: None, + has_call: false, + has_yielding_call, + op_names: Vec::new(), + terminal: JitTraceTerminal::BranchExit, + executions: 0, + ssa, + } + } + + fn tagged_side_trace_pair( + parent_frame_key: u64, + child_frame_key: u64, + parent_yields: bool, + child_yields: bool, + ) -> (TraceJitEngine, SsaExitId) { + let mut parent = SsaTraceBuilder::new(0, 1); + let parent_entry = parent.entry(); + let stack = parent + .append_param(parent_entry, SsaValueRepr::Tagged, "stack0") + .unwrap(); + let parent_exit = parent.add_exit( + 12, + vec![SsaMaterialization::Value(stack.id)], + Vec::new(), + Vec::new(), + ); + parent + .set_terminator(parent_entry, SsaTerminator::Exit { exit: parent_exit }) + .unwrap(); + + let mut child = SsaTraceBuilder::new(12, 1); + let child_entry = child.entry(); + child + .append_param(child_entry, SsaValueRepr::Tagged, "stack0") + .unwrap(); + let child_exit = child.add_exit(13, Vec::new(), Vec::new(), Vec::new()); + child + .set_terminator(child_entry, SsaTerminator::Exit { exit: child_exit }) + .unwrap(); + + let mut engine = TraceJitEngine::new(JitConfig::default()); + engine.traces.push(test_trace( + 0, + parent_frame_key, + parent_yields, + parent.finish(), + )); + engine + .traces + .push(test_trace(1, child_frame_key, child_yields, child.finish())); + (engine, parent_exit) + } + + #[test] + fn side_trace_admission_rejects_unknown_and_same_trace_ids() { + let (engine, parent_exit) = tagged_side_trace_pair(7, 7, false, false); + + assert_eq!( + engine.side_trace_import(9, parent_exit, 1), + Err(SideTraceAdmissionError::UnknownParentTrace(9)) + ); + assert_eq!( + engine.side_trace_import(0, parent_exit, 9), + Err(SideTraceAdmissionError::UnknownChildTrace(9)) + ); + assert_eq!( + engine.side_trace_import(0, SsaExitId::new(99), 1), + Err(SideTraceAdmissionError::Import( + SideTraceImportError::UnknownParentExit(SsaExitId::new(99)) + )) + ); + assert_eq!( + engine.side_trace_import(0, parent_exit, 0), + Err(SideTraceAdmissionError::SameTrace(0)) + ); + } + + #[test] + fn side_trace_admission_rejects_exit_entry_shape_mismatch() { + let (mut engine, parent_exit) = tagged_side_trace_pair(7, 7, false, false); + engine.traces[1].ssa.root_ip = 13; + + assert_eq!( + engine.side_trace_import(0, parent_exit, 1), + Err(SideTraceAdmissionError::Import( + SideTraceImportError::ExitIpMismatch { + parent: 12, + child: 13, + } + )) + ); + } + + #[test] + fn side_trace_admission_rejects_repr_mismatch_at_any_entry_param() { + let mut parent = SsaTraceBuilder::new(0, 1); + let parent_entry = parent.entry(); + let stack = parent + .append_param(parent_entry, SsaValueRepr::Tagged, "stack0") + .unwrap(); + let local = parent + .append_param(parent_entry, SsaValueRepr::I64, "local0") + .unwrap(); + let parent_exit = parent.add_exit( + 12, + vec![SsaMaterialization::Value(stack.id)], + vec![SsaMaterialization::BoxInt(local.id)], + vec![true], + ); + parent + .set_terminator(parent_entry, SsaTerminator::Exit { exit: parent_exit }) + .unwrap(); + + let mut child = SsaTraceBuilder::new(12, 1); + let child_entry = child.entry(); + child + .append_param(child_entry, SsaValueRepr::Tagged, "stack0") + .unwrap(); + child + .append_param(child_entry, SsaValueRepr::I64, "local0") + .unwrap(); + let child_exit = child.add_exit(13, Vec::new(), Vec::new(), Vec::new()); + child + .set_terminator(child_entry, SsaTerminator::Exit { exit: child_exit }) + .unwrap(); + + let mut engine = TraceJitEngine::new(JitConfig::default()); + engine.traces.push(test_trace(0, 7, false, parent.finish())); + engine.traces.push(test_trace(1, 7, false, child.finish())); + + assert_eq!( + engine.side_trace_import(0, parent_exit, 1), + Err(SideTraceAdmissionError::ReprMismatch { + index: 1, + parent: SsaValueRepr::Tagged, + child: SsaValueRepr::I64, + }) + ); + } + + #[test] + fn side_trace_admission_uses_defining_repr_for_value_materialization() { + let mut parent = SsaTraceBuilder::new(0, 1); + let parent_entry = parent.entry(); + let value = parent + .append_param(parent_entry, SsaValueRepr::I64, "stack0") + .unwrap(); + let parent_exit = parent.add_exit( + 12, + vec![SsaMaterialization::Value(value.id)], + Vec::new(), + Vec::new(), + ); + parent + .set_terminator(parent_entry, SsaTerminator::Exit { exit: parent_exit }) + .unwrap(); + + let mut child = SsaTraceBuilder::new(12, 1); + let child_entry = child.entry(); + child + .append_param(child_entry, SsaValueRepr::Tagged, "stack0") + .unwrap(); + let child_exit = child.add_exit(13, Vec::new(), Vec::new(), Vec::new()); + child + .set_terminator(child_entry, SsaTerminator::Exit { exit: child_exit }) + .unwrap(); + + let mut engine = TraceJitEngine::new(JitConfig::default()); + engine.traces.push(test_trace(0, 7, false, parent.finish())); + engine.traces.push(test_trace(1, 7, false, child.finish())); + + assert_eq!( + engine.side_trace_import(0, parent_exit, 1), + Err(SideTraceAdmissionError::ReprMismatch { + index: 0, + parent: SsaValueRepr::I64, + child: SsaValueRepr::Tagged, + }) + ); + } + + #[test] + fn side_trace_admission_rejects_box_heap_ptr_ownership() { + let mut parent = SsaTraceBuilder::new(0, 1); + let parent_entry = parent.entry(); + let heap = parent + .append_param( + parent_entry, + SsaValueRepr::HeapPtr(ValueType::String), + "stack0", + ) + .unwrap(); + let parent_exit = parent.add_exit( + 12, + vec![SsaMaterialization::BoxHeapPtr { + value: heap.id, + tag: ValueType::String, + }], + Vec::new(), + Vec::new(), + ); + parent + .set_terminator(parent_entry, SsaTerminator::Exit { exit: parent_exit }) + .unwrap(); + + let mut child = SsaTraceBuilder::new(12, 1); + let child_entry = child.entry(); + child + .append_param(child_entry, SsaValueRepr::Tagged, "stack0") + .unwrap(); + let child_exit = child.add_exit(13, Vec::new(), Vec::new(), Vec::new()); + child + .set_terminator(child_entry, SsaTerminator::Exit { exit: child_exit }) + .unwrap(); + + let mut engine = TraceJitEngine::new(JitConfig::default()); + engine.traces.push(test_trace(0, 7, false, parent.finish())); + engine.traces.push(test_trace(1, 7, false, child.finish())); + + assert_eq!( + engine.side_trace_import(0, parent_exit, 1), + Err(SideTraceAdmissionError::UnsupportedHeapOwnership { + index: 0, + value: heap.id, + }) + ); + } + + #[test] + fn side_trace_admission_rejects_yielding_parent_or_child() { + for (parent_yields, child_yields, yielding_trace_id) in [(true, false, 0), (false, true, 1)] + { + let (engine, parent_exit) = tagged_side_trace_pair(7, 7, parent_yields, child_yields); + assert_eq!( + engine.side_trace_import(0, parent_exit, 1), + Err(SideTraceAdmissionError::YieldingTrace(yielding_trace_id)) + ); + } + } + + #[test] + fn side_trace_admission_rejects_cross_frame_traces() { + let (engine, parent_exit) = tagged_side_trace_pair(7, 8, false, false); + + assert_eq!( + engine.side_trace_import(0, parent_exit, 1), + Err(SideTraceAdmissionError::CrossFrame { + parent_frame_key: 7, + child_frame_key: 8, + }) + ); + } + + #[test] + fn side_trace_admission_returns_validated_import_for_same_frame_traces() { + let mut parent = SsaTraceBuilder::new(0, 1); + let parent_entry = parent.entry(); + let stack = parent + .append_param(parent_entry, SsaValueRepr::Tagged, "stack0") + .unwrap(); + let int_local = parent + .append_param(parent_entry, SsaValueRepr::I64, "local0") + .unwrap(); + let float_local = parent + .append_param(parent_entry, SsaValueRepr::F64, "local1") + .unwrap(); + let bool_local = parent + .append_param(parent_entry, SsaValueRepr::Bool, "local2") + .unwrap(); + let parent_exit = parent.add_exit( + 12, + vec![SsaMaterialization::Value(stack.id)], + vec![ + SsaMaterialization::BoxInt(int_local.id), + SsaMaterialization::BoxFloat(float_local.id), + SsaMaterialization::BoxBool(bool_local.id), + ], + vec![true, false, true], + ); + parent + .set_terminator(parent_entry, SsaTerminator::Exit { exit: parent_exit }) + .unwrap(); + + let mut child = SsaTraceBuilder::new(12, 1); + let child_entry = child.entry(); + for label in ["stack0", "local0", "local1", "local2"] { + child + .append_param(child_entry, SsaValueRepr::Tagged, label) + .unwrap(); + } + let child_exit = child.add_exit(13, Vec::new(), Vec::new(), Vec::new()); + child + .set_terminator(child_entry, SsaTerminator::Exit { exit: child_exit }) + .unwrap(); + + let mut engine = TraceJitEngine::new(JitConfig::default()); + engine.traces.push(test_trace(0, 7, false, parent.finish())); + engine.traces.push(test_trace(1, 7, false, child.finish())); + + let import = engine.side_trace_import(0, parent_exit, 1).unwrap(); + + assert_eq!(import.parent_exit, parent_exit); + assert_eq!(import.stack_depth, 1); + assert_eq!(import.local_count, 3); + assert_eq!(import.dirty_locals, vec![true, false, true]); + assert_eq!( + import.args, + vec![ + SsaMaterialization::Value(stack.id), + SsaMaterialization::BoxInt(int_local.id), + SsaMaterialization::BoxFloat(float_local.id), + SsaMaterialization::BoxBool(bool_local.id), + ] + ); + } + + #[test] + fn region_publication_rejects_stale_trace_generation() { + let (mut engine, parent_exit) = + tagged_side_trace_pair(ROOT_FRAME_KEY, ROOT_FRAME_KEY, false, false); + let key = TraceExitKey { + parent_trace_id: 0, + exit_id: parent_exit, + }; + engine.record_trace_exit(key).unwrap(); + engine.record_trace_exit(key).unwrap(); + let candidate = engine + .region_candidate(key, 1) + .expect("same-frame candidate"); + + engine.set_config(*engine.config()); + + assert!(!engine.publish_region(&candidate)); + assert!(engine.published_region().is_none()); + } + + #[test] + fn trace_exit_profiles_are_keyed_by_parent_and_exit() { + let mut engine = TraceJitEngine::new(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 64, + }); + let mut ssa = SsaTraceBuilder::new(0, 0); + let entry = ssa.entry(); + let first_exit = ssa.add_exit(1, Vec::new(), Vec::new(), Vec::new()); + let second_exit = ssa.add_exit(2, Vec::new(), Vec::new(), Vec::new()); + ssa.set_terminator(entry, SsaTerminator::Exit { exit: first_exit }) + .unwrap(); + engine.traces.push(JitTrace { + id: 0, + frame_key: ROOT_FRAME_KEY, + root_ip: 0, + entry_stack_depth: 0, + start_line: None, + has_call: false, + has_yielding_call: false, + op_names: Vec::new(), + terminal: JitTraceTerminal::BranchExit, + executions: 0, + ssa: ssa.finish(), + }); + let first = TraceExitKey { + parent_trace_id: 0, + exit_id: first_exit, + }; + let second = TraceExitKey { + parent_trace_id: 0, + exit_id: second_exit, + }; + + engine.record_trace_exit(first).unwrap(); + engine.record_trace_exit(first).unwrap(); + engine.record_trace_exit(second).unwrap(); + + assert_eq!(engine.trace_exit_profile(first).unwrap().executions, 2); + assert_eq!(engine.trace_exit_profile(second).unwrap().executions, 1); + assert_ne!(first, second); + assert_eq!( + engine.exit_profiles(), + vec![ + JitExitProfile { + parent_trace_id: 0, + exit_id: first_exit.raw(), + executions: 2, + }, + JitExitProfile { + parent_trace_id: 0, + exit_id: second_exit.raw(), + executions: 1, + }, + ] + ); + } + + #[test] + fn trace_exit_profiles_reject_exit_missing_from_parent_trace() { + let mut engine = TraceJitEngine::new(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 64, + }); + let mut ssa = SsaTraceBuilder::new(0, 0); + let entry = ssa.entry(); + let exit_id = ssa.add_exit(1, Vec::new(), Vec::new(), Vec::new()); + ssa.set_terminator(entry, SsaTerminator::Exit { exit: exit_id }) + .unwrap(); + engine.traces.push(JitTrace { + id: 0, + frame_key: ROOT_FRAME_KEY, + root_ip: 0, + entry_stack_depth: 0, + start_line: None, + has_call: false, + has_yielding_call: false, + op_names: Vec::new(), + terminal: JitTraceTerminal::BranchExit, + executions: 0, + ssa: ssa.finish(), + }); + + assert_eq!( + engine + .record_trace_exit(TraceExitKey { + parent_trace_id: 0, + exit_id: SsaExitId::new(1), + }) + .unwrap_err(), + TraceExitRecordError::UnknownExit { + parent_trace_id: 0, + exit_id: SsaExitId::new(1), + exit_count: 1, + } + ); + assert!(engine.trace_exit_profiles.is_empty()); + } + #[test] fn scan_loop_headers_finds_backward_targets() { let mut bc = BytecodeBuilder::new(); @@ -779,6 +1637,144 @@ mod tests { assert!(!headers[branch_ip as usize]); } + #[test] + fn callable_side_exit_backoff_resets_on_native_progress() { + if !native_jit_supported() { + return; + } + let mut bc = BytecodeBuilder::new(); + let root_ip = bc.position(); + bc.ldloc(0); + bc.ldc(0); + bc.add(); + bc.stloc(0); + bc.br(root_ip); + let mut program = Program::new(vec![Value::Int(1)], bc.finish()).with_local_count(1); + program.script_functions.push(ScriptFunction { + entry_ip: root_ip, + end_ip: program.code.len() as u32, + }); + program.callable_prototypes.push(CallablePrototype { + kind: CallableKind::FunctionItem, + target: CallableTarget::ScriptFunction(0), + arity: 0, + frame_local_count: 1, + parameter_slots: Vec::new(), + capture_source_slots: Vec::new(), + capture_slots: Vec::new(), + capture_modes: Vec::new(), + self_slot: None, + schema: None, + }); + let mut engine = TraceJitEngine::new(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 64, + }); + let trace_id = engine + .observe_hot_entry(0, root_ip as usize, 0, &program) + .expect("script-frame trace should compile"); + + for _ in 1..CALLABLE_SIDE_EXIT_BACKOFF_THRESHOLD { + assert!(!engine.record_native_side_exit(trace_id)); + } + engine.record_native_loop_back(trace_id); + for _ in 1..CALLABLE_SIDE_EXIT_BACKOFF_THRESHOLD { + assert!(!engine.record_native_side_exit(trace_id)); + } + engine.record_native_region_progress(trace_id); + for _ in 1..CALLABLE_SIDE_EXIT_BACKOFF_THRESHOLD { + assert!(!engine.record_native_side_exit(trace_id)); + } + assert!(engine.record_native_side_exit(trace_id)); + engine.block_callable_frame(trace_id); + assert!(engine.callable_frame_is_blocked(0)); + assert_eq!( + engine.compiled_trace_for_entry(0, root_ip as usize, 0), + None + ); + + let root_trace_id = engine + .observe_hot_entry(ROOT_FRAME_KEY, root_ip as usize, 0, &program) + .expect("root trace should compile"); + for _ in 0..CALLABLE_SIDE_EXIT_BACKOFF_THRESHOLD * 2 { + assert!(!engine.record_native_side_exit(root_trace_id)); + } + } + + #[test] + fn callable_frame_backoff_clears_on_jit_invalidation() { + if !native_jit_supported() { + return; + } + let mut bc = BytecodeBuilder::new(); + let root_ip = bc.position(); + bc.ldloc(0); + bc.ldc(0); + bc.add(); + bc.stloc(0); + bc.br(root_ip); + let mut program = Program::new(vec![Value::Int(1)], bc.finish()).with_local_count(1); + program.script_functions.push(ScriptFunction { + entry_ip: root_ip, + end_ip: program.code.len() as u32, + }); + program.callable_prototypes.push(CallablePrototype { + kind: CallableKind::FunctionItem, + target: CallableTarget::ScriptFunction(0), + arity: 0, + frame_local_count: 1, + parameter_slots: Vec::new(), + capture_source_slots: Vec::new(), + capture_slots: Vec::new(), + capture_modes: Vec::new(), + self_slot: None, + schema: None, + }); + let config = JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 64, + }; + let mut engine = TraceJitEngine::new(config); + + let trace_id = engine + .observe_hot_entry(0, root_ip as usize, 0, &program) + .expect("script-frame trace should compile"); + for _ in 0..CALLABLE_SIDE_EXIT_BACKOFF_THRESHOLD { + engine.record_native_side_exit(trace_id); + } + engine.block_callable_frame(trace_id); + assert!(engine.callable_frame_is_blocked(0)); + + engine.reset_runtime_backoff(); + assert!(!engine.callable_frame_is_blocked(0)); + assert_eq!( + engine.compiled_trace_for_entry(0, root_ip as usize, 0), + Some(trace_id) + ); + for _ in 0..CALLABLE_SIDE_EXIT_BACKOFF_THRESHOLD { + engine.record_native_side_exit(trace_id); + } + engine.block_callable_frame(trace_id); + assert!(engine.callable_frame_is_blocked(0)); + + engine.set_config(config); + assert!(!engine.callable_frame_is_blocked(0)); + + let trace_id = engine + .observe_hot_entry(0, root_ip as usize, 0, &program) + .expect("script-frame trace should recompile after config invalidation"); + for _ in 0..CALLABLE_SIDE_EXIT_BACKOFF_THRESHOLD { + engine.record_native_side_exit(trace_id); + } + engine.block_callable_frame(trace_id); + assert!(engine.callable_frame_is_blocked(0)); + + assert!(engine.set_non_yielding_host_imports(vec![true])); + assert!(!engine.callable_frame_is_blocked(0)); + } + #[test] fn trace_entry_cache_separates_root_and_script_frames() { if !native_jit_supported() { diff --git a/src/vm/mod.rs b/src/vm/mod.rs index b9e9bf51..07135f9a 100644 --- a/src/vm/mod.rs +++ b/src/vm/mod.rs @@ -359,6 +359,16 @@ pub struct Vm { jit: jit::TraceJitEngine, native_traces: Vec>, native_trace_exec_count: u64, + jit_native_region_entry_count: u64, + jit_native_region_edge_count: u64, + jit_native_direct_link_count: u64, + jit_native_direct_links_enabled: bool, + jit_native_direct_cross_frame_enabled: bool, + jit_native_active_direct_trace_id: usize, + jit_native_direct_escape_streak: u16, + jit_native_direct_region_fallback: bool, + jit_native_compile_time_ns: u64, + jit_native_region_compile_time_ns: u64, jit_trace_exit_count: u64, jit_native_loop_back_count: u64, jit_native_link_handoff_count: u64, @@ -678,6 +688,16 @@ impl Vm { jit: jit::TraceJitEngine::new(jit_config), native_traces: Vec::new(), native_trace_exec_count: 0, + jit_native_region_entry_count: 0, + jit_native_region_edge_count: 0, + jit_native_direct_link_count: 0, + jit_native_direct_links_enabled: true, + jit_native_direct_cross_frame_enabled: false, + jit_native_active_direct_trace_id: usize::MAX, + jit_native_direct_escape_streak: 0, + jit_native_direct_region_fallback: false, + jit_native_compile_time_ns: 0, + jit_native_region_compile_time_ns: 0, jit_trace_exit_count: 0, jit_native_loop_back_count: 0, jit_native_link_handoff_count: 0, @@ -908,6 +928,7 @@ impl Vm { self.waiting_host_op = None; self.io_state = crate::builtins::runtime::IoState::default(); self.map_iterators.clear(); + self.jit.reset_runtime_backoff(); self.clear_interpreter_metrics(); } @@ -1094,6 +1115,7 @@ impl Vm { pub fn set_drop_contract_events_enabled(&mut self, enabled: bool) { if self.drop_contract_events_enabled != enabled { + self.disconnect_native_regions(); self.native_traces.clear(); } self.drop_contract_events_enabled = enabled; @@ -2148,10 +2170,12 @@ impl Vm { && !self.drop_contract_events_enabled() { let frame_key = self.active_frame_key(); - let stack_depth = self.active_operand_stack_len(); - let entry_local_types = (frame_key != crate::vm::native::ROOT_FRAME_KEY) - .then(|| self.active_local_types()); - let trace_id = { + let trace_id = if self.jit.callable_frame_is_blocked(frame_key) { + None + } else { + let stack_depth = self.active_operand_stack_len(); + let entry_local_types = (frame_key != crate::vm::native::ROOT_FRAME_KEY) + .then(|| self.active_local_types()); let program = &self.program; self.jit.observe_hot_entry_with_local_types( frame_key, diff --git a/src/vm/native/bridge.rs b/src/vm/native/bridge.rs index e34dd2c9..7be6bd25 100644 --- a/src/vm/native/bridge.rs +++ b/src/vm/native/bridge.rs @@ -16,6 +16,20 @@ pub(crate) const STATUS_WAITING: i32 = 4; pub(crate) const STATUS_OUT_OF_FUEL: i32 = 5; pub(crate) const STATUS_LINKED_CONTINUE: i32 = 6; pub(crate) const STATUS_ERROR: i32 = -1; +pub(crate) const STATUS_JIT_TRACE_EXIT_BASE: i32 = 0x100; +pub(crate) const STATUS_JIT_TRACE_EXIT_MAX_ID: u32 = u16::MAX as u32; +pub(crate) const STATUS_JIT_TRACE_EXIT_MAX: i32 = + STATUS_JIT_TRACE_EXIT_BASE + STATUS_JIT_TRACE_EXIT_MAX_ID as i32; + +pub(crate) fn encode_jit_trace_exit_status(exit_id: u32) -> Option { + (exit_id <= STATUS_JIT_TRACE_EXIT_MAX_ID).then(|| STATUS_JIT_TRACE_EXIT_BASE + exit_id as i32) +} + +pub(crate) fn decode_jit_trace_exit_status(status: i32) -> Option { + (STATUS_JIT_TRACE_EXIT_BASE..=STATUS_JIT_TRACE_EXIT_MAX) + .contains(&status) + .then(|| (status - STATUS_JIT_TRACE_EXIT_BASE) as u32) +} pub(crate) const ROOT_FRAME_KEY: u64 = u64::MAX; @@ -121,7 +135,7 @@ unsafe fn arc_from_repr_ptr(ptr: *mut u8) -> Arc { unsafe { std::mem::transmute_copy(&ptr) } } -fn run_step(vm: *mut Vm, helper_name: &str, f: F) -> i32 +fn run_step(vm: *mut Vm, helper_name: &'static str, f: F) -> i32 where F: FnOnce(&mut Vm) -> VmResult, { @@ -132,6 +146,7 @@ where return STATUS_ERROR; }; + vm_ref.record_native_bridge_hit(helper_name); match f(vm_ref) { Ok(status) => status, Err(err) => { @@ -285,10 +300,18 @@ pub(crate) fn enter_call_value_entry_address() -> usize { pd_vm_native_enter_call_value as *const () as usize } +pub(crate) fn enter_call_value_inherited_entry_address() -> usize { + pd_vm_native_enter_call_value_inherited as *const () as usize +} + pub(crate) fn leave_frame_entry_address() -> usize { pd_vm_native_leave_frame as *const () as usize } +pub(crate) fn leave_frame_inherited_entry_address() -> usize { + pd_vm_native_leave_frame_inherited as *const () as usize +} + pub(crate) fn active_local_base_entry_address() -> usize { pd_vm_native_active_local_base as *const () as usize } @@ -345,6 +368,14 @@ pub(crate) fn non_yielding_host_call_entry_address() -> usize { pd_vm_native_non_yielding_host_call as *const () as usize } +pub(crate) fn non_yielding_scalar_host_call_entry_address() -> usize { + pd_vm_native_non_yielding_scalar_host_call as *const () as usize +} + +pub(crate) fn non_yielding_i64_host_call_entry_address() -> usize { + pd_vm_native_non_yielding_i64_host_call as *const () as usize +} + pub(crate) fn helper_entry_offset() -> i32 { i32::try_from(std::mem::offset_of!(Vm, native_helper_fn)) .expect("Vm::native_helper_fn offset must fit i32") @@ -752,6 +783,118 @@ pub(crate) extern "C" fn pd_vm_native_restore_exit_state( }) } +fn native_frame_state(vm: &Vm) -> VmResult { + let frame = vm.execution_frames.last(); + let operand_stack_base = frame.map(|frame| frame.operand_stack_base).unwrap_or(0); + let local_base = frame.map(|frame| frame.local_base).unwrap_or(0); + let local_count = frame + .map(|frame| frame.local_count) + .unwrap_or(vm.locals.len()); + let active_stack_len = vm + .stack + .len() + .checked_sub(operand_stack_base) + .ok_or_else(|| { + VmError::JitNative("native frame-state stack base exceeds stack length".to_string()) + })?; + let frame_key = frame + .and_then(|frame| frame.prototype_id) + .map(u64::from) + .unwrap_or(ROOT_FRAME_KEY); + let continuation_kind = match frame.map(|frame| &frame.continuation) { + Some(FrameContinuation::Halt) | None => 0, + Some(FrameContinuation::ResumeBytecode { .. }) => 1, + Some(FrameContinuation::ReturnToHost) => 2, + }; + Ok(NativeFrameState { + frame_key, + operand_stack_base, + active_stack_len, + local_base, + local_count, + frame_depth: vm.call_depth, + continuation_kind, + }) +} + +fn write_inherited_state_packet(vm: &Vm, packet: *mut u8) -> VmResult<()> { + use super::{ + INHERITED_STATE_ACTIVE_OFFSET, INHERITED_STATE_DYNAMIC_TARGET_OFFSET, + INHERITED_STATE_FRAME_KEY_OFFSET, INHERITED_STATE_LOCAL_BASE_OFFSET, + INHERITED_STATE_STACK_BASE_OFFSET, INHERITED_STATE_TARGET_IP_OFFSET, + INHERITED_STATE_VALUE_COUNT_OFFSET, INHERITED_STATE_VALUES_OFFSET, + MAX_INHERITED_ENTRY_VALUES, + }; + + if packet.is_null() { + return Err(VmError::JitNative( + "native inherited-state packet pointer is null".to_string(), + )); + } + let state = native_frame_state(vm)?; + let value_count = state + .active_stack_len + .checked_add(state.local_count) + .ok_or_else(|| VmError::JitNative("native inherited value count overflow".to_string()))?; + unsafe { + packet + .add(INHERITED_STATE_ACTIVE_OFFSET as usize) + .cast::() + .write(0); + packet + .add(INHERITED_STATE_DYNAMIC_TARGET_OFFSET as usize) + .cast::() + .write(0); + } + if value_count > MAX_INHERITED_ENTRY_VALUES { + return Ok(()); + } + unsafe { + packet + .add(INHERITED_STATE_FRAME_KEY_OFFSET as usize) + .cast::() + .write(state.frame_key); + packet + .add(INHERITED_STATE_STACK_BASE_OFFSET as usize) + .cast::() + .write(state.operand_stack_base); + packet + .add(INHERITED_STATE_LOCAL_BASE_OFFSET as usize) + .cast::() + .write(state.local_base); + packet + .add(INHERITED_STATE_TARGET_IP_OFFSET as usize) + .cast::() + .write(vm.ip); + packet + .add(INHERITED_STATE_VALUE_COUNT_OFFSET as usize) + .cast::() + .write(value_count); + let values = packet + .add(INHERITED_STATE_VALUES_OFFSET as usize) + .cast::<*const Value>(); + let stack = vm.stack.as_ptr().add(state.operand_stack_base); + for index in 0..state.active_stack_len { + values.add(index).write(stack.add(index)); + } + let locals = vm.locals.as_ptr().add(state.local_base); + for index in 0..state.local_count { + values + .add(state.active_stack_len + index) + .write(locals.add(index)); + } + packet + .add(INHERITED_STATE_DYNAMIC_TARGET_OFFSET as usize) + .cast::() + .write(vm.jit_native_inherited_target()); + packet + .add(INHERITED_STATE_ACTIVE_OFFSET as usize) + .cast::() + .write(1); + } + Ok(()) +} + pub(crate) extern "C" fn pd_vm_native_frame_state(vm: *mut Vm, out: *mut NativeFrameState) -> i32 { run_step(vm, "frame_state", |vm| { if out.is_null() { @@ -759,43 +902,45 @@ pub(crate) extern "C" fn pd_vm_native_frame_state(vm: *mut Vm, out: *mut NativeF "native frame-state helper received null output".to_string(), )); } - let frame = vm.execution_frames.last(); - let operand_stack_base = frame.map(|frame| frame.operand_stack_base).unwrap_or(0); - let local_base = frame.map(|frame| frame.local_base).unwrap_or(0); - let local_count = frame - .map(|frame| frame.local_count) - .unwrap_or(vm.locals.len()); - let active_stack_len = vm - .stack - .len() - .checked_sub(operand_stack_base) - .ok_or_else(|| { - VmError::JitNative("native frame-state stack base exceeds stack length".to_string()) - })?; - let frame_key = frame - .and_then(|frame| frame.prototype_id) - .map(u64::from) - .unwrap_or(ROOT_FRAME_KEY); - let continuation_kind = match frame.map(|frame| &frame.continuation) { - Some(FrameContinuation::Halt) | None => 0, - Some(FrameContinuation::ResumeBytecode { .. }) => 1, - Some(FrameContinuation::ReturnToHost) => 2, - }; unsafe { - out.write(NativeFrameState { - frame_key, - operand_stack_base, - active_stack_len, - local_base, - local_count, - frame_depth: vm.call_depth, - continuation_kind, - }); + out.write(native_frame_state(vm)?); } Ok(STATUS_CONTINUE) }) } +fn native_enter_call_value( + vm: &mut Vm, + argc: i64, + call_ip: i64, + resume_ip: i64, + inherited_state: *mut u8, +) -> VmResult { + let argc = u8::try_from(argc) + .map_err(|_| VmError::InvalidFrameState("native call argc out of range"))?; + let call_ip = usize::try_from(call_ip) + .map_err(|_| VmError::InvalidFrameState("native call ip out of range"))?; + let resume_ip = usize::try_from(resume_ip) + .map_err(|_| VmError::InvalidFrameState("native resume ip out of range"))?; + if vm.ip != call_ip { + vm.jump_to(call_ip)?; + } + if resume_ip > vm.program.code.len() { + return Err(VmError::BytecodeBounds); + } + vm.ip = resume_ip; + let status = match vm.execute_call_value(argc)? { + ExecOutcome::Continue => STATUS_LINKED_CONTINUE, + ExecOutcome::Halted => STATUS_HALTED, + ExecOutcome::Yielded => STATUS_YIELDED, + ExecOutcome::Waiting(_) => STATUS_WAITING, + }; + if status == STATUS_LINKED_CONTINUE && !inherited_state.is_null() { + write_inherited_state_packet(vm, inherited_state)?; + } + Ok(status) +} + pub(crate) extern "C" fn pd_vm_native_enter_call_value( vm: *mut Vm, argc: i64, @@ -803,39 +948,51 @@ pub(crate) extern "C" fn pd_vm_native_enter_call_value( resume_ip: i64, ) -> i32 { run_step(vm, "enter_call_value", |vm| { - let argc = u8::try_from(argc) - .map_err(|_| VmError::InvalidFrameState("native call argc out of range"))?; - let call_ip = usize::try_from(call_ip) - .map_err(|_| VmError::InvalidFrameState("native call ip out of range"))?; - let resume_ip = usize::try_from(resume_ip) - .map_err(|_| VmError::InvalidFrameState("native resume ip out of range"))?; - if vm.ip != call_ip { - vm.jump_to(call_ip)?; - } - if resume_ip > vm.program.code.len() { - return Err(VmError::BytecodeBounds); - } - vm.ip = resume_ip; - match vm.execute_call_value(argc)? { - ExecOutcome::Continue => Ok(STATUS_LINKED_CONTINUE), - ExecOutcome::Halted => Ok(STATUS_HALTED), - ExecOutcome::Yielded => Ok(STATUS_YIELDED), - ExecOutcome::Waiting(_) => Ok(STATUS_WAITING), - } + native_enter_call_value(vm, argc, call_ip, resume_ip, std::ptr::null_mut()) }) } +pub(crate) extern "C" fn pd_vm_native_enter_call_value_inherited( + vm: *mut Vm, + argc: i64, + call_ip: i64, + resume_ip: i64, + inherited_state: *mut u8, +) -> i32 { + run_step(vm, "enter_call_value", |vm| { + native_enter_call_value(vm, argc, call_ip, resume_ip, inherited_state) + }) +} + +fn native_leave_frame(vm: &mut Vm, ret_ip: i64, inherited_state: *mut u8) -> VmResult { + let ret_ip = usize::try_from(ret_ip) + .map_err(|_| VmError::InvalidFrameState("native ret ip out of range"))?; + vm.jump_to(ret_ip)?; + let status = match vm.complete_active_frame()? { + ExecOutcome::Continue => STATUS_LINKED_CONTINUE, + ExecOutcome::Halted => STATUS_HALTED, + ExecOutcome::Yielded => STATUS_YIELDED, + ExecOutcome::Waiting(_) => STATUS_WAITING, + }; + if status == STATUS_LINKED_CONTINUE && !inherited_state.is_null() { + write_inherited_state_packet(vm, inherited_state)?; + } + Ok(status) +} + pub(crate) extern "C" fn pd_vm_native_leave_frame(vm: *mut Vm, ret_ip: i64) -> i32 { run_step(vm, "leave_frame", |vm| { - let ret_ip = usize::try_from(ret_ip) - .map_err(|_| VmError::InvalidFrameState("native ret ip out of range"))?; - vm.jump_to(ret_ip)?; - match vm.complete_active_frame()? { - ExecOutcome::Continue => Ok(STATUS_LINKED_CONTINUE), - ExecOutcome::Halted => Ok(STATUS_HALTED), - ExecOutcome::Yielded => Ok(STATUS_YIELDED), - ExecOutcome::Waiting(_) => Ok(STATUS_WAITING), - } + native_leave_frame(vm, ret_ip, std::ptr::null_mut()) + }) +} + +pub(crate) extern "C" fn pd_vm_native_leave_frame_inherited( + vm: *mut Vm, + ret_ip: i64, + inherited_state: *mut u8, +) -> i32 { + run_step(vm, "leave_frame", |vm| { + native_leave_frame(vm, ret_ip, inherited_state) }) } @@ -1308,28 +1465,148 @@ pub(crate) extern "C" fn pd_vm_native_non_yielding_host_call( )); return STATUS_ERROR; } - let Some(&resolved) = vm.resolved_calls.get(import) else { - store_bridge_error(VmError::InvalidCall(import as u16)); - return STATUS_ERROR; + let args = unsafe { std::slice::from_raw_parts(args, argc) }; + let expected_return_type = vm + .program + .imports + .get(import) + .map(|host_import| host_import.return_type); + match call_non_yielding_host_value(vm, import, args, expected_return_type) { + Ok(value) => { + unsafe { std::ptr::write(out, value) }; + STATUS_CONTINUE + } + Err(err) => { + store_bridge_error(err); + STATUS_ERROR + } + } +} + +fn call_non_yielding_host_value( + vm: &mut Vm, + import: usize, + args: &[Value], + expected_return_type: Option, +) -> VmResult { + let resolved = *vm + .resolved_calls + .get(import) + .ok_or(VmError::InvalidCall(import as u16))?; + let function = match vm.host_functions.get(usize::from(resolved)) { + Some(VmHostFunction::ArgsStaticNonYielding(function)) => *function, + _ => { + return Err(VmError::JitNative( + "native host-call binding changed after trace compilation".to_string(), + )); + } + }; + vm.call_depth = vm.call_depth.saturating_add(1); + let outcome = function(args); + vm.call_depth = vm.call_depth.saturating_sub(1); + outcome + .and_then(crate::vm::host::require_non_yielding_host_value) + .and_then(|value| { + crate::vm::host::validate_non_yielding_host_value(value, expected_return_type) + }) +} + +fn scalar_host_return_type(return_type: i64) -> VmResult { + if return_type == ValueType::Int as i64 { + Ok(ValueType::Int) + } else if return_type == ValueType::Float as i64 { + Ok(ValueType::Float) + } else if return_type == ValueType::Bool as i64 { + Ok(ValueType::Bool) + } else { + Err(VmError::JitNative( + "native scalar host-call return type is unsupported".to_string(), + )) + } +} + +fn store_scalar_host_result(value: Value, return_type: i64, out: *mut u64) -> VmResult<()> { + let bits = match (return_type, value) { + (tag, Value::Int(value)) if tag == ValueType::Int as i64 => value as u64, + (tag, Value::Float(value)) if tag == ValueType::Float as i64 => value.to_bits(), + (tag, Value::Bool(value)) if tag == ValueType::Bool as i64 => u64::from(value), + (tag, _) if tag == ValueType::Int as i64 => return Err(VmError::TypeMismatch("int")), + (tag, _) if tag == ValueType::Float as i64 => { + return Err(VmError::TypeMismatch("float")); + } + (tag, _) if tag == ValueType::Bool as i64 => return Err(VmError::TypeMismatch("bool")), + _ => { + return Err(VmError::JitNative( + "native scalar host-call return type is unsupported".to_string(), + )); + } }; - let Some(VmHostFunction::ArgsStaticNonYielding(function)) = - vm.host_functions.get(usize::from(resolved)) - else { + unsafe { out.write(bits) }; + Ok(()) +} + +pub(crate) extern "C" fn pd_vm_native_non_yielding_scalar_host_call( + vm: *mut Vm, + import: usize, + args: *const Value, + argc: usize, + return_type: i64, + out: *mut u64, +) -> i32 { + let Some(vm) = (unsafe { vm.as_mut() }) else { store_bridge_error(VmError::JitNative( - "native host-call binding changed after trace compilation".to_string(), + "native scalar host-call helper received null vm pointer".to_string(), )); return STATUS_ERROR; }; - + if out.is_null() || (argc != 0 && args.is_null()) || argc > u8::MAX as usize { + store_bridge_error(VmError::JitNative( + "native scalar host-call helper received invalid storage".to_string(), + )); + return STATUS_ERROR; + } let args = unsafe { std::slice::from_raw_parts(args, argc) }; - vm.call_depth = vm.call_depth.saturating_add(1); - let outcome = function(args); - vm.call_depth = vm.call_depth.saturating_sub(1); - match outcome.and_then(crate::vm::host::require_non_yielding_host_value) { - Ok(value) => { - unsafe { std::ptr::write(out, value) }; - STATUS_CONTINUE + match scalar_host_return_type(return_type) + .and_then(|expected| call_non_yielding_host_value(vm, import, args, Some(expected))) + .and_then(|value| store_scalar_host_result(value, return_type, out)) + { + Ok(()) => STATUS_CONTINUE, + Err(err) => { + store_bridge_error(err); + STATUS_ERROR } + } +} + +pub(crate) extern "C" fn pd_vm_native_non_yielding_i64_host_call( + vm: *mut Vm, + import: usize, + arg0: i64, + arg1: i64, + argc: usize, + return_type: i64, + out: *mut u64, +) -> i32 { + let Some(vm) = (unsafe { vm.as_mut() }) else { + store_bridge_error(VmError::JitNative( + "native i64 host-call helper received null vm pointer".to_string(), + )); + return STATUS_ERROR; + }; + if out.is_null() || argc > 2 { + store_bridge_error(VmError::JitNative( + "native i64 host-call helper received invalid storage".to_string(), + )); + return STATUS_ERROR; + } + let storage = [Value::Int(arg0), Value::Int(arg1)]; + match scalar_host_return_type(return_type) + .and_then(|expected| { + call_non_yielding_host_value(vm, import, &storage[..argc], Some(expected)) + }) + .and_then(|value| store_scalar_host_result(value, return_type, out)) + { + Ok(()) => STATUS_CONTINUE, Err(err) => { store_bridge_error(err); STATUS_ERROR @@ -1557,6 +1834,30 @@ mod tests { use super::*; use std::mem::{ManuallyDrop, MaybeUninit}; + #[test] + fn jit_trace_exit_status_round_trips_reserved_range_boundaries() { + for exit_id in [0, 1, 7, 255, STATUS_JIT_TRACE_EXIT_MAX_ID] { + let status = encode_jit_trace_exit_status(exit_id).unwrap(); + assert_eq!(decode_jit_trace_exit_status(status), Some(exit_id)); + assert_ne!(status, STATUS_TRACE_EXIT); + assert_ne!(status, STATUS_YIELDED); + assert_ne!(status, STATUS_WAITING); + assert_ne!(status, STATUS_OUT_OF_FUEL); + assert_ne!(status, STATUS_LINKED_CONTINUE); + } + assert_eq!( + encode_jit_trace_exit_status(STATUS_JIT_TRACE_EXIT_MAX_ID + 1), + None + ); + assert_eq!(decode_jit_trace_exit_status(STATUS_TRACE_EXIT), None); + assert_eq!(decode_jit_trace_exit_status(STATUS_ERROR), None); + assert_eq!( + decode_jit_trace_exit_status(STATUS_JIT_TRACE_EXIT_MAX + 1), + None + ); + assert_eq!(decode_jit_trace_exit_status(i32::MAX), None); + } + #[test] fn native_frame_state_and_active_restore_are_frame_relative() { let program = diff --git a/src/vm/native/codegen.rs b/src/vm/native/codegen.rs index fe96bad4..d3decbb7 100644 --- a/src/vm/native/codegen.rs +++ b/src/vm/native/codegen.rs @@ -50,6 +50,16 @@ pub(crate) fn enter_call_value_signature( sig } +#[cfg(feature = "cranelift-jit")] +pub(crate) fn enter_call_value_inherited_signature( + pointer_type: cranelift_codegen::ir::Type, + call_conv: cranelift_codegen::isa::CallConv, +) -> Signature { + let mut sig = enter_call_value_signature(pointer_type, call_conv); + sig.params.push(AbiParam::new(pointer_type)); + sig +} + #[cfg(feature = "cranelift-jit")] pub(crate) fn leave_frame_signature( pointer_type: cranelift_codegen::ir::Type, @@ -62,6 +72,16 @@ pub(crate) fn leave_frame_signature( sig } +#[cfg(feature = "cranelift-jit")] +pub(crate) fn leave_frame_inherited_signature( + pointer_type: cranelift_codegen::ir::Type, + call_conv: cranelift_codegen::isa::CallConv, +) -> Signature { + let mut sig = leave_frame_signature(pointer_type, call_conv); + sig.params.push(AbiParam::new(pointer_type)); + sig +} + #[cfg(feature = "cranelift-jit")] pub(crate) fn frame_state_signature( pointer_type: cranelift_codegen::ir::Type, @@ -212,6 +232,36 @@ pub(crate) fn non_yielding_host_call_signature( sig } +#[cfg(feature = "cranelift-jit")] +pub(crate) fn non_yielding_scalar_host_call_signature( + pointer_type: cranelift_codegen::ir::Type, + call_conv: cranelift_codegen::isa::CallConv, +) -> Signature { + let mut sig = Signature::new(call_conv); + for _ in 0..6 { + sig.params.push(AbiParam::new(pointer_type)); + } + sig.returns.push(AbiParam::new(types::I32)); + sig +} + +#[cfg(feature = "cranelift-jit")] +pub(crate) fn non_yielding_i64_host_call_signature( + pointer_type: cranelift_codegen::ir::Type, + call_conv: cranelift_codegen::isa::CallConv, +) -> Signature { + let mut sig = Signature::new(call_conv); + sig.params.push(AbiParam::new(pointer_type)); + sig.params.push(AbiParam::new(pointer_type)); + sig.params.push(AbiParam::new(types::I64)); + sig.params.push(AbiParam::new(types::I64)); + sig.params.push(AbiParam::new(pointer_type)); + sig.params.push(AbiParam::new(types::I64)); + sig.params.push(AbiParam::new(pointer_type)); + sig.returns.push(AbiParam::new(types::I32)); + sig +} + #[cfg(feature = "cranelift-jit")] pub(crate) fn collection_predicate_signature( pointer_type: cranelift_codegen::ir::Type, diff --git a/src/vm/native/exec.rs b/src/vm/native/exec.rs index 6bf454f6..48bb260a 100644 --- a/src/vm/native/exec.rs +++ b/src/vm/native/exec.rs @@ -25,7 +25,10 @@ impl ExecutableBuffer { std::ptr::copy_nonoverlapping(code.as_ptr(), ptr, code.len()); }); flush_instruction_cache(ptr, code.len()); - prepare_for_execution(); + if let Err(err) = protect_executable_buffer(ptr, code.len()) { + free_executable(ptr, code.len()); + return Err(err); + } } Ok(Self { @@ -67,7 +70,7 @@ impl Drop for ExecutableBuffer { #[cfg(windows)] unsafe fn alloc_executable(len: usize) -> VmResult<*mut u8> { use windows_sys::Win32::System::Memory::{ - MEM_COMMIT, MEM_RESERVE, PAGE_EXECUTE_READWRITE, VirtualAlloc, + MEM_COMMIT, MEM_RESERVE, PAGE_READWRITE, VirtualAlloc, }; let ptr = unsafe { @@ -75,7 +78,7 @@ unsafe fn alloc_executable(len: usize) -> VmResult<*mut u8> { std::ptr::null_mut(), len, MEM_COMMIT | MEM_RESERVE, - PAGE_EXECUTE_READWRITE, + PAGE_READWRITE, ) } as *mut u8; if ptr.is_null() { @@ -104,11 +107,15 @@ unsafe fn flush_instruction_cache(ptr: *mut u8, len: usize) { #[cfg(unix)] unsafe fn alloc_executable(len: usize) -> VmResult<*mut u8> { + #[cfg(all(target_os = "macos", target_arch = "aarch64"))] + let prot = libc::PROT_READ | libc::PROT_WRITE | libc::PROT_EXEC; + #[cfg(not(all(target_os = "macos", target_arch = "aarch64")))] + let prot = libc::PROT_READ | libc::PROT_WRITE; let ptr = unsafe { libc::mmap( std::ptr::null_mut(), len, - libc::PROT_READ | libc::PROT_WRITE | libc::PROT_EXEC, + prot, libc::MAP_PRIVATE | map_anon_flag() | map_jit_flag(), -1, 0, @@ -173,6 +180,41 @@ const fn map_jit_flag() -> i32 { 0 } +#[cfg(all(unix, target_os = "macos", target_arch = "aarch64"))] +pub(crate) unsafe fn protect_executable_buffer(_ptr: *mut u8, _len: usize) -> VmResult<()> { + if unsafe { libc::pthread_jit_write_protect_supported_np() } == 0 { + return Err(VmError::JitNative( + "macOS JIT write protection is unavailable".to_string(), + )); + } + unsafe { libc::pthread_jit_write_protect_np(1) }; + Ok(()) +} + +#[cfg(all(unix, not(all(target_os = "macos", target_arch = "aarch64"))))] +pub(crate) unsafe fn protect_executable_buffer(ptr: *mut u8, len: usize) -> VmResult<()> { + if unsafe { libc::mprotect(ptr.cast(), len, libc::PROT_READ | libc::PROT_EXEC) } != 0 { + return Err(VmError::JitNative(format!( + "mprotect failed for executable trace buffer: {}", + std::io::Error::last_os_error() + ))); + } + Ok(()) +} + +#[cfg(windows)] +pub(crate) unsafe fn protect_executable_buffer(ptr: *mut u8, len: usize) -> VmResult<()> { + use windows_sys::Win32::System::Memory::{PAGE_EXECUTE_READ, VirtualProtect}; + + let mut previous = 0; + if unsafe { VirtualProtect(ptr.cast(), len, PAGE_EXECUTE_READ, &mut previous) } == 0 { + return Err(VmError::JitNative( + "VirtualProtect failed for executable trace buffer".to_string(), + )); + } + Ok(()) +} + #[cfg(all(unix, target_os = "macos", target_arch = "aarch64"))] pub(crate) unsafe fn prepare_for_execution() { if unsafe { libc::pthread_jit_write_protect_supported_np() } != 0 { diff --git a/src/vm/native/layout.rs b/src/vm/native/layout.rs index 0c84a938..c41c95a5 100644 --- a/src/vm/native/layout.rs +++ b/src/vm/native/layout.rs @@ -48,6 +48,9 @@ pub(crate) struct NativeStackLayout { pub(crate) vm_fuel_ops_until_check_offset: i32, pub(crate) vm_epoch_deadline_offset: i32, pub(crate) vm_epoch_counter_ptr_offset: i32, + pub(crate) vm_jit_native_region_edge_count_offset: i32, + pub(crate) vm_jit_native_direct_link_count_offset: i32, + pub(crate) vm_jit_native_active_direct_trace_id_offset: i32, pub(crate) stack_vec: VecLayout, pub(crate) map: MapLayout, pub(crate) value: ValueLayout, @@ -86,6 +89,18 @@ fn detect_native_stack_layout_uncached() -> VmResult { std::mem::offset_of!(Vm, epoch_counter_ptr), "Vm::epoch_counter_ptr offset", )?; + let vm_jit_native_region_edge_count_offset = usize_to_i32( + std::mem::offset_of!(Vm, jit_native_region_edge_count), + "Vm::jit_native_region_edge_count offset", + )?; + let vm_jit_native_direct_link_count_offset = usize_to_i32( + std::mem::offset_of!(Vm, jit_native_direct_link_count), + "Vm::jit_native_direct_link_count offset", + )?; + let vm_jit_native_active_direct_trace_id_offset = usize_to_i32( + std::mem::offset_of!(Vm, jit_native_active_direct_trace_id), + "Vm::jit_native_active_direct_trace_id offset", + )?; let stack_vec = detect_vec_layout()?; let map = detect_map_layout()?; let value = detect_value_layout()?; @@ -98,6 +113,9 @@ fn detect_native_stack_layout_uncached() -> VmResult { vm_fuel_ops_until_check_offset, vm_epoch_deadline_offset, vm_epoch_counter_ptr_offset, + vm_jit_native_region_edge_count_offset, + vm_jit_native_direct_link_count_offset, + vm_jit_native_active_direct_trace_id_offset, stack_vec, map, value, diff --git a/src/vm/native/mod.rs b/src/vm/native/mod.rs index 22245541..49b87e34 100644 --- a/src/vm/native/mod.rs +++ b/src/vm/native/mod.rs @@ -7,36 +7,40 @@ mod offsets; pub(crate) use bridge::{ NativeFrameState, NativeInterruptMode, NativeInterruptSettings, OP_BUILTIN_CALL, OP_CALL, - ROOT_FRAME_KEY, STATUS_CONTINUE, STATUS_ERROR, STATUS_HALTED, STATUS_LINKED_CONTINUE, - STATUS_OUT_OF_FUEL, STATUS_TRACE_EXIT, STATUS_WAITING, STATUS_YIELDED, + ROOT_FRAME_KEY, STATUS_CONTINUE, STATUS_ERROR, STATUS_HALTED, STATUS_JIT_TRACE_EXIT_BASE, + STATUS_LINKED_CONTINUE, STATUS_OUT_OF_FUEL, STATUS_TRACE_EXIT, STATUS_WAITING, STATUS_YIELDED, active_local_base_entry_address, active_stack_base_entry_address, alloc_byte_buffer_entry_address, alloc_value_buffer_entry_address, aot_call_boundary_interrupt_entry_address, array_push_entry_address, array_set_entry_address, clear_bridge_error, clear_value_slot_entry_address, clone_value_to_slot_entry_address, - collection_set_entry_address, copy_bytes_entry_address, enter_call_value_entry_address, - frame_state_entry_address, helper_entry_address, helper_entry_offset, - init_null_value_slot_entry_address, interrupt_helper_entry_address, - interrupt_helper_entry_offset, leave_frame_entry_address, map_get_entry_address, - map_has_entry_address, map_iter_next_entry_address, map_iter_take_key_entry_address, - map_iter_take_value_entry_address, map_set_entry_address, non_yielding_host_call_entry_address, - regex_match_entry_address, regex_replace_entry_address, - restore_active_exit_state_entry_address, restore_active_sparse_exit_state_entry_address, - restore_exit_state_entry_address, restore_sparse_exit_state_entry_address, - shared_array_from_buffer_entry_address, shared_bytes_from_buffer_entry_address, - shared_string_from_buffer_entry_address, store_bridge_error, string_contains_entry_address, - string_lower_ascii_entry_address, string_replace_literal_entry_address, - string_split_literal_entry_address, take_bridge_error, to_string_entry_address, - type_of_entry_address, value_eq_entry_address, value_len_entry_address, - write_heap_value_to_slot_entry_address, zero_bytes_entry_address, + collection_set_entry_address, copy_bytes_entry_address, decode_jit_trace_exit_status, + encode_jit_trace_exit_status, enter_call_value_entry_address, + enter_call_value_inherited_entry_address, frame_state_entry_address, helper_entry_address, + helper_entry_offset, init_null_value_slot_entry_address, interrupt_helper_entry_address, + interrupt_helper_entry_offset, leave_frame_entry_address, leave_frame_inherited_entry_address, + map_get_entry_address, map_has_entry_address, map_iter_next_entry_address, + map_iter_take_key_entry_address, map_iter_take_value_entry_address, map_set_entry_address, + non_yielding_host_call_entry_address, non_yielding_i64_host_call_entry_address, + non_yielding_scalar_host_call_entry_address, regex_match_entry_address, + regex_replace_entry_address, restore_active_exit_state_entry_address, + restore_active_sparse_exit_state_entry_address, restore_exit_state_entry_address, + restore_sparse_exit_state_entry_address, shared_array_from_buffer_entry_address, + shared_bytes_from_buffer_entry_address, shared_string_from_buffer_entry_address, + store_bridge_error, string_contains_entry_address, string_lower_ascii_entry_address, + string_replace_literal_entry_address, string_split_literal_entry_address, take_bridge_error, + to_string_entry_address, type_of_entry_address, value_eq_entry_address, + value_len_entry_address, write_heap_value_to_slot_entry_address, zero_bytes_entry_address, }; #[cfg(feature = "cranelift-jit")] pub(crate) use codegen::{ alloc_buffer_signature, array_set_signature, box_heap_value_signature, clone_value_signature, collection_get_signature, collection_mutation_signature, collection_predicate_signature, - copy_bytes_signature, enter_call_value_signature, entry_signature, frame_state_signature, - free_buffer_signature, helper_signature, jump_with_status, leave_frame_signature, + copy_bytes_signature, enter_call_value_inherited_signature, enter_call_value_signature, + entry_signature, frame_state_signature, free_buffer_signature, helper_signature, + jump_with_status, leave_frame_inherited_signature, leave_frame_signature, map_iter_next_signature, map_iter_take_signature, map_set_signature, - non_yielding_host_call_signature, pack_shared_signature, regex_match_signature, + non_yielding_host_call_signature, non_yielding_i64_host_call_signature, + non_yielding_scalar_host_call_signature, pack_shared_signature, regex_match_signature, regex_replace_signature, restore_exit_signature, sparse_restore_exit_signature, string_binary_transform_signature, string_contains_signature, string_replace_signature, string_unary_transform_signature, value_eq_signature, value_len_signature, @@ -50,6 +54,15 @@ pub(crate) use layout::{ pub(crate) use offsets::{HeapIntrinsicAddrs, HeapIntrinsicRefs, ResolvedOffsets, resolve_offsets}; pub(crate) const NATIVE_CALLABLE_ABI_VERSION: u16 = 4; +pub(crate) const MAX_INHERITED_ENTRY_VALUES: usize = 256; +pub(crate) const INHERITED_STATE_ACTIVE_OFFSET: i32 = 0; +pub(crate) const INHERITED_STATE_FRAME_KEY_OFFSET: i32 = 8; +pub(crate) const INHERITED_STATE_STACK_BASE_OFFSET: i32 = 16; +pub(crate) const INHERITED_STATE_LOCAL_BASE_OFFSET: i32 = 24; +pub(crate) const INHERITED_STATE_DYNAMIC_TARGET_OFFSET: i32 = 32; +pub(crate) const INHERITED_STATE_TARGET_IP_OFFSET: i32 = 40; +pub(crate) const INHERITED_STATE_VALUE_COUNT_OFFSET: i32 = 48; +pub(crate) const INHERITED_STATE_VALUES_OFFSET: i32 = 56; #[cfg(feature = "cranelift-jit")] pub(crate) fn selected_codegen_backend() -> &'static str { diff --git a/tests/jit/jit_tests.rs b/tests/jit/jit_tests.rs index a2325fca..9ea4b236 100644 --- a/tests/jit/jit_tests.rs +++ b/tests/jit/jit_tests.rs @@ -1,3 +1,5 @@ +use std::{cell::Cell, sync::Arc}; + use vm::{ BytecodeBuilder, CallOutcome, CallReturn, HostFunction, JitConfig, JitTraceTerminal, OpCode, Program, Value, ValueType, Vm, VmStatus, VmYieldReason, builtin_call_index, compile_source, @@ -11,6 +13,32 @@ fn native_jit_supported() -> bool { && (cfg!(target_os = "linux") || cfg!(target_os = "macos"))) } +fn is_jit_state_boundary_bridge(name: &str) -> bool { + matches!( + name, + "frame_state" + | "leave_frame" + | "restore_active_exit_state" + | "restore_active_sparse_exit_state" + | "restore_exit_state" + | "restore_sparse_exit_state" + ) +} + +#[test] +fn jit_snapshot_literal_preserves_public_source_shape() { + let snapshot = vm::jit::JitSnapshot { + arch: "test", + config: JitConfig::default(), + traces: Vec::new(), + attempts: Vec::new(), + metrics: vm::jit::JitMetrics::default(), + nyi_reference: Vec::new(), + }; + + assert!(snapshot.traces.is_empty()); +} + fn any_trace_op(snapshot: &vm::jit::JitSnapshot, op: &str) -> bool { snapshot .traces @@ -345,7 +373,9 @@ fn aot_inlines_typed_numeric_steps_without_bridge_fallback() { assert_eq!(vm.stack(), &[Value::Float(3.375)]); let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(name, _)| *name == "ldc"), + bridge_hits + .iter() + .all(|(name, _)| *name == "ldc" || is_jit_state_boundary_bridge(name)), "typed aot arithmetic should only fall back for initial stack growth, not math ops: {bridge_hits:?}" ); } @@ -1719,6 +1749,212 @@ fn increment_non_yielding(args: &[Value]) -> Result { Ok(CallOutcome::Return(CallReturn::one(Value::Int(value + 1)))) } +fn less_non_yielding(args: &[Value]) -> Result { + let [Value::Int(lhs), Value::Int(rhs)] = args else { + return Err(vm::VmError::TypeMismatch("two ints")); + }; + Ok(CallOutcome::Return(CallReturn::one(Value::Bool(lhs < rhs)))) +} + +fn string_len_non_yielding(args: &[Value]) -> Result { + let [Value::String(value)] = args else { + return Err(vm::VmError::TypeMismatch("string")); + }; + Ok(CallOutcome::Return(CallReturn::one(Value::Int( + value.len() as i64, + )))) +} + +fn mixed_float_non_yielding(args: &[Value]) -> Result { + let [ + Value::Int(integer), + Value::Float(float), + Value::Bool(boolean), + ] = args + else { + return Err(vm::VmError::TypeMismatch("int, float, bool")); + }; + Ok(CallOutcome::Return(CallReturn::one(Value::Float( + *integer as f64 + float + if *boolean { 1.0 } else { 0.0 }, + )))) +} + +thread_local! { + static UNSTABLE_HOST_RETURN_COUNT: Cell = const { Cell::new(0) }; +} + +fn unstable_return_non_yielding(_: &[Value]) -> Result { + let call = UNSTABLE_HOST_RETURN_COUNT.with(|count| { + let call = count.get(); + count.set(call.saturating_add(1)); + call + }); + let value = if call < 8 { + Value::Int(1) + } else { + Value::Bool(false) + }; + Ok(CallOutcome::Return(CallReturn::one(value))) +} + +#[test] +fn trace_jit_enforces_scalar_host_return_contract_after_dirty_local_write() { + if !native_jit_supported() { + return; + } + UNSTABLE_HOST_RETURN_COUNT.with(|count| count.set(0)); + let compiled = compile_source( + r#" + fn unstable() -> int; + let mut dirty = 0; + while dirty < 100 { + dirty = dirty + 1; + dirty = dirty + unstable(); + } + dirty; + "#, + ) + .expect("compile should succeed"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + vm.bind_static_non_yielding_args_function("unstable", unstable_return_non_yielding); + + assert!(matches!(vm.run(), Err(vm::VmError::TypeMismatch("int")))); + assert!( + vm.jit_native_exec_count() > 0, + "return mismatch should occur after native execution:\n{}", + vm.dump_jit_info() + ); +} + +#[test] +fn trace_jit_passes_mixed_host_args_and_float_return_as_scalars() { + if !native_jit_supported() { + return; + } + let compiled = compile_source( + r#" + fn mixed(integer: int, float: float, boolean: bool) -> float; + let mut i = 0; + let mut total = 0.0; + while i < 100 { + total = total + mixed(i, 0.5, true); + i = i + 1; + } + total; + "#, + ) + .expect("compile should succeed"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + vm.bind_static_non_yielding_args_function("mixed", mixed_float_non_yielding); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Float(5_100.0)]); + assert!( + vm.jit_snapshot().traces.iter().any(|trace| { + trace + .ssa_text() + .lines() + .any(|line| line.contains(":f64 = host_call")) + }), + "mixed host arguments should return an SSA float:\n{}", + vm.dump_jit_info() + ); + assert!(vm.jit_native_exec_count() > 0); +} + +#[test] +fn trace_jit_passes_tagged_host_args_to_scalar_return() { + if !native_jit_supported() { + return; + } + let compiled = compile_source( + r#" + fn string_len(value: string) -> int; + let mut i = 0; + while i < 100 { + i = i + string_len("x"); + } + i; + "#, + ) + .expect("compile should succeed"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + vm.bind_static_non_yielding_args_function("string_len", string_len_non_yielding); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(100)]); + assert!( + vm.jit_snapshot().traces.iter().any(|trace| { + trace + .ssa_text() + .lines() + .any(|line| line.contains(":i64 = host_call")) + }), + "tagged argument host call should return an SSA scalar:\n{}", + vm.dump_jit_info() + ); + assert!(vm.jit_native_exec_count() > 0); +} + +#[test] +fn trace_jit_passes_i64_host_args_and_bool_return_as_scalars() { + if !native_jit_supported() { + return; + } + let compiled = compile_source( + r#" + fn less(lhs: int, rhs: int) -> bool; + let mut i = 0; + let mut matched = 0; + while i < 100 { + if less(i, 50) { + matched = matched + 1; + } + i = i + 1; + } + matched; + "#, + ) + .expect("compile should succeed"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + vm.bind_static_non_yielding_args_function("less", less_non_yielding); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(50)]); + let snapshot = vm.jit_snapshot(); + assert!( + snapshot.traces.iter().any(|trace| { + trace + .ssa_text() + .lines() + .any(|line| line.contains(":bool = host_call")) + }), + "bool host return should remain an SSA scalar:\n{}", + vm.dump_jit_info() + ); + assert!(vm.jit_native_exec_count() > 0); +} + #[test] fn trace_jit_keeps_non_yielding_static_args_calls_inside_loop_trace() { if !native_jit_supported() { @@ -1752,14 +1988,34 @@ fn trace_jit_keeps_non_yielding_static_args_calls_inside_loop_trace() { assert_eq!(status.unwrap(), VmStatus::Halted); assert_eq!(vm.stack(), &[Value::Int(100)]); let snapshot = vm.jit_snapshot(); - assert!( - snapshot.traces.iter().any(|trace| { + let host_trace = snapshot + .traces + .iter() + .find(|trace| { trace.terminal == JitTraceTerminal::LoopBack && trace.op_names().iter().any(|op| op == "host_call") && trace.ssa_text().contains("host_call") - }), - "non-yielding call should remain inside a loop-back trace, dump:\n{}", - vm.dump_jit_info() + }) + .unwrap_or_else(|| { + panic!( + "non-yielding call should remain inside a loop-back trace, dump:\n{}", + vm.dump_jit_info() + ) + }); + let host_ssa = host_trace.ssa_text(); + assert!( + host_ssa + .lines() + .any(|line| line.contains(":i64 = host_call")), + "typed host return should remain an unboxed SSA scalar:\n{host_ssa}" + ); + let host_result_tail = host_ssa + .split_once("host_call") + .map(|(_, tail)| tail) + .expect("host-call SSA line should exist"); + assert!( + !host_result_tail.contains("unbox_int"), + "typed host result should not pass through tagged return storage:\n{host_ssa}" ); assert!(vm.jit_native_exec_count() > 0); } @@ -2179,96 +2435,930 @@ fn trace_jit_links_between_nested_loop_native_traces() { } #[test] -fn trace_jit_restores_tagged_heap_locals_on_ssa_exit() { +fn trace_jit_reports_exact_parent_exit_profiles() { if !native_jit_supported() { return; } + let source = r#" + fn choose(i) { + let mut result = 7; + if i % 2 == 0 { + result = 3; + } + if i % 3 == 1 { + result = 5; + } + result + } - let mut bc = BytecodeBuilder::new(); - bc.ldc(0); - bc.stloc(0); - bc.ldc(1); - bc.stloc(1); - bc.ldc(2); - bc.stloc(2); - let root_ip = bc.position(); - bc.ldloc(2); - bc.ldc(4); - bc.clt(); - let guard_ip = bc.position(); - bc.brfalse(0); - bc.ldloc(0); - bc.stloc(1); - bc.ldloc(2); - bc.ldc(3); - bc.add(); - bc.stloc(2); - bc.br(root_ip); - let exit_ip = bc.position(); - bc.ldloc(1); - bc.ret(); - - let mut code = bc.finish(); - patch_branch_target(&mut code, guard_ip, exit_ip); - let program = Program::new( - vec![ - Value::string("seed"), - Value::string(""), - Value::Int(0), - Value::Int(1), - Value::Int(4), - ], - code, - ) - .with_local_count(3); - - let mut vm = Vm::new(program); + let mut i = 0; + let mut total = 0; + while i < 64 { + total = total + choose(i); + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).expect("branch profile fixture should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); vm.set_jit_config(JitConfig { enabled: true, hot_loop_threshold: 1, - max_trace_len: 512, + max_trace_len: 256, }); - let status = vm.run().expect("vm should run"); - assert_eq!(status, VmStatus::Halted); - assert_eq!(vm.stack(), &[Value::string("seed")]); - assert!( - vm.jit_native_exec_count() > 0, - "expected native SSA execution for tagged-local rebinding loop, dump:\n{}", - vm.dump_jit_info() + assert_eq!( + vm.run().expect("branch profile fixture should run"), + VmStatus::Halted ); + assert_eq!(vm.stack(), &[Value::Int(318)]); - let dump = vm.dump_jit_info(); let snapshot = vm.jit_snapshot(); + let exit_profiles = vm.jit_exit_profiles(); assert!( - dump.contains("lowering=ssa"), - "expected tagged-local rebinding loop to lower through SSA, dump:\n{dump}" + !exit_profiles.is_empty(), + "expected exact parent-exit profiles:\n{}", + vm.dump_jit_info() ); - assert_eq!( - snapshot.metrics.helper_fallback_count, 0, - "tagged-local rebinding loop should not need interpreter fallback, dump:\n{dump}" + for profile in &exit_profiles { + let parent = &snapshot.traces[profile.parent_trace_id]; + assert!(profile.exit_id < parent.ssa_exit_count() as u32); + assert!(profile.executions > 0); + } + assert!( + snapshot + .traces + .iter() + .enumerate() + .filter(|(_, trace)| trace.frame_key != u64::MAX) + .any(|(parent_trace_id, _)| { + let parent_profiles = exit_profiles + .iter() + .filter(|profile| profile.parent_trace_id == parent_trace_id) + .collect::>(); + parent_profiles.len() >= 2 + && parent_profiles.iter().any(|profile| profile.executions > 1) + }), + "expected two distinct exits and one repeated exact exit under a callable parent:\n{}", + vm.dump_jit_info() ); } #[test] -fn trace_jit_restores_array_and_map_locals_on_ssa_exit() { +fn trace_jit_direct_side_link_bypasses_rust_dispatch() { if !native_jit_supported() { return; } + let source = r#" + let mut i = 0; + let mut total = 0; + while i < 4096 { + if i % 2 == 0 { + total = total + 3; + } else { + total = total + 5; + } + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).expect("direct side-link fixture should compile"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_native_direct_links_enabled(true); + vm.set_jit_native_bridge_stats_enabled(true); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); - let array_value = Value::array(vec![Value::Int(1), Value::Int(2), Value::Int(3)]); - let map_value = Value::map(vec![(Value::string("k"), Value::Int(9))]); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(16_384)]); + let first_direct_links = vm.jit_native_direct_link_count(); + let first_handoffs = vm.jit_native_link_handoff_count(); + let first_bridge_stats = vm.jit_native_bridge_stats_snapshot(); + assert!(first_direct_links > 4_000, "{}", vm.dump_jit_info()); + assert_eq!(vm.jit_native_region_count(), 0, "{}", vm.dump_jit_info()); + assert!( + first_bridge_stats + .iter() + .any(|(name, count)| *name == "frame_state" && *count > 0), + "direct-link characterization must count native frame-state bridges: {first_bridge_stats:?}" + ); + assert!( + first_bridge_stats + .iter() + .any(|(name, count)| { *name == "restore_active_sparse_exit_state" && *count > 0 }), + "direct-link characterization must count sparse exit restores: {first_bridge_stats:?}" + ); - let mut bc = BytecodeBuilder::new(); - bc.ldc(0); - bc.stloc(0); - bc.ldc(1); - bc.stloc(1); - bc.ldc(2); - bc.stloc(2); - let root_ip = bc.position(); - bc.ldloc(2); - bc.ldc(4); + vm.clear_jit_native_bridge_stats(); + vm.reset_for_reuse(); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(16_384)]); + assert!( + vm.jit_native_direct_link_count() - first_direct_links > 4_000, + "{}", + vm.dump_jit_info() + ); + assert!( + vm.jit_native_link_handoff_count() - first_handoffs <= 2, + "{}", + vm.dump_jit_info() + ); + let steady_bridge_stats = vm.jit_native_bridge_stats_snapshot(); + let steady_frame_state = steady_bridge_stats + .iter() + .find_map(|(name, count)| (*name == "frame_state").then_some(*count)) + .unwrap_or(0); + let steady_sparse_restores = steady_bridge_stats + .iter() + .find_map(|(name, count)| (*name == "restore_active_sparse_exit_state").then_some(*count)) + .unwrap_or(0); + assert!( + steady_frame_state <= 4, + "direct-linked child entries must inherit values without reloading VM frame state: \ + frame_state={steady_frame_state} stats={steady_bridge_stats:?}\n{}", + vm.dump_jit_info() + ); + assert!( + steady_sparse_restores <= 2, + "direct-linked edges must skip sparse VM restoration: \ + restores={steady_sparse_restores} stats={steady_bridge_stats:?}\n{}", + vm.dump_jit_info() + ); +} + +#[test] +fn trace_jit_tail_link_cycle_has_bounded_host_stack() { + if !native_jit_supported() { + return; + } + let source = r#" + let mut i = 0; + let mut total = 0; + while i < 1000000 { + if i % 2 == 0 { + total = total + 3; + } else { + total = total + 5; + } + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).expect("tail-link cycle fixture should compile"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_native_direct_links_enabled(true); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(4_000_000)]); + assert!( + vm.jit_native_direct_link_count() > 999_000, + "{}", + vm.dump_jit_info() + ); + assert!( + vm.jit_native_link_handoff_count() <= 4, + "{}", + vm.dump_jit_info() + ); +} + +#[test] +fn trace_jit_side_link_invalidation_clears_incoming_slots() { + if !native_jit_supported() { + return; + } + let source = r#" + let mut i = 0; + let mut total = 0; + while i < 4096 { + if i % 2 == 0 { total = total + 3; } else { total = total + 5; } + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).expect("side-link invalidation fixture should compile"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_native_direct_links_enabled(true); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert!(vm.jit_native_active_direct_link_slot_count() > 0); + + vm.set_jit_native_direct_links_enabled(false); + assert_eq!(vm.jit_native_active_direct_link_slot_count(), 0); + assert_eq!(vm.jit_native_trace_count(), 0); +} + +#[test] +fn trace_jit_side_link_generation_prevents_stale_entry_reuse() { + if !native_jit_supported() { + return; + } + let source = r#" + let mut i = 0; + let mut total = 0; + while i < 4096 { + if i % 2 == 0 { total = total + 3; } else { total = total + 5; } + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).expect("side-link generation fixture should compile"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_native_direct_links_enabled(true); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert!(vm.jit_native_active_direct_link_slot_count() > 0); + + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 257, + }); + assert_eq!(vm.jit_native_active_direct_link_slot_count(), 0); + assert_eq!(vm.jit_native_direct_link_count(), 0); + vm.reset_for_reuse(); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(16_384)]); + assert!(vm.jit_native_direct_link_count() > 4_000); + assert!(vm.jit_native_active_direct_link_slot_count() > 0); +} + +#[test] +fn trace_jit_side_link_respects_callable_frame_and_interrupt_boundaries() { + if !native_jit_supported() { + return; + } + let source = r#" + fn run(limit) { + let mut i = 0; + let mut total = 0; + while i < limit { + if i % 2 == 0 { total = total + 3; } else { total = total + 5; } + i = i + 1; + } + total + } + run(4096); + "#; + let compiled = compile_source(source).expect("direct boundary fixture should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(true); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + vm.set_fuel_check_interval(1).unwrap(); + vm.set_fuel(1_000_000); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(16_384)]); + assert!(vm.jit_native_direct_link_count() > 4_000); + + vm.reset_for_reuse(); + vm.set_fuel(8); + let mut fuel_yields = 0_u64; + loop { + match vm.run().expect("direct-link fuel run should make progress") { + VmStatus::Halted => break, + VmStatus::Yielded => { + fuel_yields = fuel_yields.saturating_add(1); + assert_eq!(vm.last_yield_reason(), Some(VmYieldReason::Fuel)); + assert!(fuel_yields < 4_096, "{}", vm.dump_jit_info()); + vm.recharge_fuel(128).unwrap(); + } + VmStatus::Waiting(op_id) => panic!("unexpected host wait {op_id}"), + } + } + assert_eq!(vm.stack(), &[Value::Int(16_384)]); + assert!(fuel_yields > 0); + assert!(vm.jit_native_direct_link_count() > 4_000); +} + +#[test] +fn trace_jit_region_links_hot_same_frame_side_exit() { + if !native_jit_supported() { + return; + } + let source = r#" + let mut i = 0; + let mut total = 0; + while i < 256 { + if i % 2 == 0 { + total = total + 3; + } else { + total = total + 5; + } + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).expect("region fixture should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(false); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + + assert_eq!(vm.run().expect("first region run"), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(1_024)]); + let first_handoffs = vm.jit_native_link_handoff_count(); + let first_native_execs = vm.jit_native_exec_count(); + let first_region_entries = vm.jit_native_region_entry_count(); + let first_internal_edges = vm.jit_native_internal_region_edge_count(); + let first_fallbacks = vm.jit_helper_fallback_count(); + assert!( + first_handoffs > 0, + "expected warmup to cross native trace boundaries:\n{}", + vm.dump_jit_info() + ); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + assert!(first_region_entries > 0, "{}", vm.dump_jit_info()); + assert!(first_internal_edges > 0, "{}", vm.dump_jit_info()); + + vm.reset_for_reuse(); + assert_eq!(vm.run().expect("second region run"), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(1_024)]); + + let second_run_handoffs = vm + .jit_native_link_handoff_count() + .saturating_sub(first_handoffs); + assert!( + second_run_handoffs <= 2, + "expected cyclic same-frame region fusion to bound external handoffs, second_run_handoffs={second_run_handoffs}:\n{}", + vm.dump_jit_info() + ); + assert!(vm.jit_native_exec_count() > first_native_execs); + assert!(vm.jit_native_region_entry_count() > first_region_entries); + assert!(vm.jit_native_internal_region_edge_count() > first_internal_edges); + assert_eq!(vm.jit_helper_fallback_count(), first_fallbacks); +} + +#[test] +fn trace_jit_region_cycle_propagates_disjoint_dirty_locals() { + if !native_jit_supported() { + return; + } + let source = r#" + fn branch_counts(limit) { + let mut i = 0; + let mut even = 0; + let mut odd = 0; + while i < limit { + if i % 2 == 0 { + even = even + 1; + } else { + odd = odd + 1; + } + i = i + 1; + } + even * 10000 + odd + } + branch_counts(4096); + "#; + let compiled = compile_source(source).expect("disjoint dirty region compile should succeed"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_native_direct_links_enabled(false); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + + assert_eq!( + vm.run().expect("first disjoint dirty region run"), + VmStatus::Halted + ); + assert_eq!( + vm.stack(), + &[Value::Int(20_482_048)], + "{}", + vm.dump_jit_info() + ); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + + vm.reset_for_reuse(); + assert_eq!( + vm.run().expect("second disjoint dirty region run"), + VmStatus::Halted + ); + assert_eq!( + vm.stack(), + &[Value::Int(20_482_048)], + "{}", + vm.dump_jit_info() + ); +} + +#[test] +fn trace_jit_region_unlinked_exit_restores_callable_frame() { + if !native_jit_supported() { + return; + } + let source = r#" + fn probe(limit, stop) { + let mut i = 0; + let mut total = 0; + while i < limit { + if i == stop { + break; + } + if i % 2 == 0 { + total = total + 3; + } else { + total = total + 5; + } + i = i + 1; + } + total * 1000 + i + } + probe(257, 50) + 1; + "#; + let compiled = compile_source(source).expect("unlinked callable exit fixture should compile"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_native_direct_links_enabled(false); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(200_051)], "{}", vm.dump_jit_info()); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + assert!(vm.jit_native_internal_region_edge_count() > 0); + assert_eq!(vm.jit_helper_fallback_count(), 0, "{}", vm.dump_jit_info()); +} + +#[test] +fn trace_jit_region_preserves_owned_value_drop_contract() { + if !native_jit_supported() { + return; + } + let source = r#" + let payload = ["owned"]; + let mut i = 0; + let mut total = 0; + while i < 50 { + if i % 2 == 0 { + total = total + 3; + } else { + total = total + 5; + } + i = i + 1; + } + [payload, total]; + "#; + let compiled = compile_source(source).expect("owned region fixture should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(false); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + let Value::Array(output) = &vm.stack()[0] else { + panic!("expected owned array output: {}", vm.dump_jit_info()); + }; + let output = Arc::clone(output); + assert_eq!(output.as_slice()[1], Value::Int(200)); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + assert!(vm.jit_native_internal_region_edge_count() > 0); + assert_eq!(Arc::strong_count(&output), 2); + + vm.set_drop_contract_events_enabled(true); + assert_eq!(vm.jit_native_region_count(), 0); + vm.reset_for_reuse(); + assert_eq!(Arc::strong_count(&output), 1); + drop(vm); + assert_eq!(Arc::strong_count(&output), 1); +} + +#[test] +fn trace_jit_region_progress_prevents_callable_frame_backoff() { + if !native_jit_supported() { + return; + } + let source = r#" + fn run(limit, payload) { + let mut i = 0; + let mut total = 0; + while i < limit { + if i % 2 == 0 { + total = total + 3; + } else { + total = total + 5; + } + i = i + 1; + } + total + payload[0] + } + run(256, [1]); + "#; + let compiled = compile_source(source).expect("region progress fixture should compile"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_native_direct_links_enabled(false); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(1_025)]); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + let first_execs = vm.jit_native_exec_count(); + let first_edges = vm.jit_native_internal_region_edge_count(); + + vm.reset_for_reuse(); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(1_025)]); + assert!( + vm.jit_native_exec_count() > first_execs, + "{}", + vm.dump_jit_info() + ); + assert!( + vm.jit_native_internal_region_edge_count() > first_edges, + "{}", + vm.dump_jit_info() + ); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); +} + +#[test] +fn trace_jit_inherited_direct_progress_prevents_callable_frame_backoff() { + if !native_jit_supported() { + return; + } + let source = r#" + fn run(limit, payload) { + let mut i = 0; + let mut total = 0; + while i < limit { + if i % 2 == 0 { + total = total + 3; + } else { + total = total + 5; + } + i = i + 1; + } + total + payload[0] + } + let mut rounds = 0; + let mut sum = 0; + while rounds < 32 { + sum = sum + run(64, [1]); + rounds = rounds + 1; + } + sum; + "#; + let compiled = compile_source(source).expect("direct progress fixture should compile"); + let mut vm = Vm::new(compiled.program); + vm.set_jit_native_direct_links_enabled(true); + vm.set_jit_native_bridge_stats_enabled(true); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(8_224)]); + assert!( + vm.jit_native_active_direct_link_slot_count() > 0, + "{}", + vm.dump_jit_info() + ); + let first_direct_links = vm.jit_native_direct_link_count(); + + vm.clear_jit_native_bridge_stats(); + vm.reset_for_reuse(); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(8_224)]); + assert!( + vm.jit_native_direct_link_count() - first_direct_links > 500, + "{}", + vm.dump_jit_info() + ); + let bridge_stats = vm.jit_native_bridge_stats_snapshot(); + let frame_state = bridge_stats + .iter() + .find_map(|(name, count)| (*name == "frame_state").then_some(*count)) + .unwrap_or(0); + assert!( + frame_state <= 40, + "callable direct graph must keep inherited state across escaped side paths: \ + frame_state={frame_state} stats={bridge_stats:?}\n{}", + vm.dump_jit_info() + ); +} + +#[test] +fn trace_jit_region_respects_fuel_and_epoch_interrupts() { + if !native_jit_supported() { + return; + } + let source = r#" + let mut i = 0; + let mut total = 0; + while i < 256 { + if i % 2 == 0 { + total = total + 3; + } else { + total = total + 5; + } + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).expect("region interrupt fixture should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(false); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + + vm.set_fuel_check_interval(1).unwrap(); + vm.set_fuel(1_000_000); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + + vm.reset_for_reuse(); + vm.set_fuel(8); + let mut fuel_yields = 0_u64; + loop { + match vm.run().expect("fuel region run should make progress") { + VmStatus::Halted => break, + VmStatus::Yielded => { + fuel_yields = fuel_yields.saturating_add(1); + assert_eq!(vm.last_yield_reason(), Some(VmYieldReason::Fuel)); + assert!(fuel_yields < 1_024, "{}", vm.dump_jit_info()); + vm.recharge_fuel(16).unwrap(); + } + VmStatus::Waiting(op_id) => panic!("unexpected host wait {op_id}"), + } + } + assert!(fuel_yields > 0); + assert_eq!(vm.stack(), &[Value::Int(1_024)]); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + + vm.clear_fuel(); + vm.reset_for_reuse(); + vm.set_epoch_check_interval(1).unwrap(); + vm.set_epoch_deadline(1_000_000).unwrap(); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + + vm.reset_for_reuse(); + vm.set_epoch_deadline(0).unwrap(); + assert_eq!(vm.run().unwrap(), VmStatus::Yielded); + assert_eq!(vm.last_yield_reason(), Some(VmYieldReason::Epoch)); + vm.clear_epoch_deadline(); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(1_024)]); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); +} + +#[test] +fn trace_jit_region_reports_compile_and_code_telemetry() { + if !native_jit_supported() { + return; + } + let source = r#" + let mut i = 0; + let mut total = 0; + while i < 128 { + if i % 2 == 0 { + total = total + 3; + } else { + total = total + 5; + } + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).expect("region telemetry fixture should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(false); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert!(vm.jit_native_code_bytes() > 0, "{}", vm.dump_jit_info()); + assert!( + vm.jit_native_region_code_bytes() > 0, + "{}", + vm.dump_jit_info() + ); + assert!( + vm.jit_native_compile_time_ns() > 0, + "{}", + vm.dump_jit_info() + ); + assert!( + vm.jit_native_region_compile_time_ns() > 0, + "{}", + vm.dump_jit_info() + ); +} + +#[test] +fn trace_jit_region_republishes_after_native_settings_change() { + if !native_jit_supported() { + return; + } + let source = r#" + let mut i = 0; + let mut total = 0; + while i < 256 { + if i % 2 == 0 { + total = total + 3; + } else { + total = total + 5; + } + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).expect("region fixture should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(false); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + vm.set_fuel_check_interval(64).unwrap(); + vm.set_fuel(1_000_000); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + let first_edges = vm.jit_native_internal_region_edge_count(); + + vm.reset_for_reuse(); + vm.set_fuel_check_interval(1).unwrap(); + vm.set_fuel(1_000_000); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(1_024)]); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + assert!(vm.jit_native_internal_region_edge_count() > first_edges); +} + +#[test] +fn trace_jit_region_invalidation_releases_owner_and_can_republish() { + if !native_jit_supported() { + return; + } + let source = r#" + let mut i = 0; + let mut total = 0; + while i < 256 { + if i % 2 == 0 { + total = total + 3; + } else { + total = total + 5; + } + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).expect("region fixture should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(false); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + + vm.set_drop_contract_events_enabled(true); + assert_eq!(vm.jit_native_region_count(), 0); + vm.set_drop_contract_events_enabled(false); + vm.reset_for_reuse(); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(1_024)]); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); +} + +#[test] +fn trace_jit_restores_tagged_heap_locals_on_ssa_exit() { + if !native_jit_supported() { + return; + } + + let mut bc = BytecodeBuilder::new(); + bc.ldc(0); + bc.stloc(0); + bc.ldc(1); + bc.stloc(1); + bc.ldc(2); + bc.stloc(2); + let root_ip = bc.position(); + bc.ldloc(2); + bc.ldc(4); + bc.clt(); + let guard_ip = bc.position(); + bc.brfalse(0); + bc.ldloc(0); + bc.stloc(1); + bc.ldloc(2); + bc.ldc(3); + bc.add(); + bc.stloc(2); + bc.br(root_ip); + let exit_ip = bc.position(); + bc.ldloc(1); + bc.ret(); + + let mut code = bc.finish(); + patch_branch_target(&mut code, guard_ip, exit_ip); + let program = Program::new( + vec![ + Value::string("seed"), + Value::string(""), + Value::Int(0), + Value::Int(1), + Value::Int(4), + ], + code, + ) + .with_local_count(3); + + let mut vm = Vm::new(program); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + + let status = vm.run().expect("vm should run"); + assert_eq!(status, VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::string("seed")]); + assert!( + vm.jit_native_exec_count() > 0, + "expected native SSA execution for tagged-local rebinding loop, dump:\n{}", + vm.dump_jit_info() + ); + + let dump = vm.dump_jit_info(); + let snapshot = vm.jit_snapshot(); + assert!( + dump.contains("lowering=ssa"), + "expected tagged-local rebinding loop to lower through SSA, dump:\n{dump}" + ); + assert_eq!( + snapshot.metrics.helper_fallback_count, 0, + "tagged-local rebinding loop should not need interpreter fallback, dump:\n{dump}" + ); +} + +#[test] +fn trace_jit_restores_array_and_map_locals_on_ssa_exit() { + if !native_jit_supported() { + return; + } + + let array_value = Value::array(vec![Value::Int(1), Value::Int(2), Value::Int(3)]); + let map_value = Value::map(vec![(Value::string("k"), Value::Int(9))]); + + let mut bc = BytecodeBuilder::new(); + bc.ldc(0); + bc.stloc(0); + bc.ldc(1); + bc.stloc(1); + bc.ldc(2); + bc.stloc(2); + let root_ip = bc.position(); + bc.ldloc(2); + bc.ldc(4); bc.clt(); let guard_ip = bc.position(); bc.brfalse(0); @@ -2361,9 +3451,9 @@ fn trace_jit_supports_array_len_get_has_in_ssa() { let status = vm.run().expect("array trace vm should run"); assert_eq!(status, VmStatus::Halted); - assert_eq!(vm.stack(), &[Value::Int(60)]); - let dump = vm.dump_jit_info(); + assert_eq!(vm.stack(), &[Value::Int(60)], "{dump}"); + let snapshot = vm.jit_snapshot(); assert!( vm.jit_native_exec_count() > 0, @@ -3055,7 +4145,9 @@ fn trace_jit_supports_float_math_in_ssa() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "float loop should not use native helper bridges, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -3099,7 +4191,9 @@ fn trace_jit_supports_string_call_boundary_exits() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "string concat loop should not need native helper bridges, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -3144,7 +4238,9 @@ fn trace_jit_supports_bytes_call_boundary_exits() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "bytes concat loop should not need native helper bridges for call-boundary execution, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -3203,7 +4299,9 @@ fn trace_jit_supports_bytes_sequence_call_boundary_exits() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "bytes builtin loop should not need native helper bridges for call-boundary execution, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -3258,7 +4356,9 @@ fn trace_jit_supports_string_sequence_call_boundary_exits() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "string builtin loop should not need native helper bridges for call-boundary execution, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -3394,7 +4494,9 @@ fn trace_jit_supports_bytes_len_get_slice_in_ssa() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "bytes builtin loop should not need native helper bridges for call-boundary execution, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -3493,7 +4595,9 @@ fn trace_jit_supports_bytes_has_in_ssa() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "bytes has loop should not need native helper bridges, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -3740,7 +4844,9 @@ fn trace_jit_supports_string_len_get_slice_in_ssa() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "string builtin loop should not need native helper bridges for call-boundary execution, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -3833,7 +4939,9 @@ fn trace_jit_supports_manual_string_concat_in_ssa() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "manual string concat loop should not need native helper bridges for call-boundary execution, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -3931,7 +5039,9 @@ fn trace_jit_supports_manual_bytes_concat_in_ssa() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "manual bytes concat loop should not need native helper bridges for call-boundary execution, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -4044,12 +5154,135 @@ fn trace_jit_supports_bytes_array_codecs_in_ssa() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "bytes array codec loop should not need native helper bridges, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); } +#[test] +fn trace_jit_specializes_ascii_bytes_to_utf8_in_ssa() { + if !native_jit_supported() { + return; + } + + let from_call = + builtin_call_index("bytes::from_array_u8").expect("bytes::from_array_u8 should exist"); + let utf8_call = builtin_call_index("bytes::to_utf8").expect("bytes::to_utf8 should exist"); + let mut bc = BytecodeBuilder::new(); + for (constant, local) in [(0, 0), (1, 1), (2, 2), (3, 3)] { + bc.ldc(constant); + bc.stloc(local); + } + let root_ip = bc.position(); + bc.ldloc(3); + bc.ldc(4); + let clt_ip = bc.position(); + bc.clt(); + let guard_ip = bc.position(); + bc.brfalse(0); + bc.ldloc(0); + let from_ip = bc.position(); + bc.call(from_call, 1); + bc.stloc(1); + bc.ldloc(1); + let utf8_ip = bc.position(); + bc.call(utf8_call, 1); + bc.stloc(2); + bc.ldloc(3); + bc.ldc(5); + let add_ip = bc.position(); + bc.add(); + bc.stloc(3); + bc.br(root_ip); + let exit_ip = bc.position(); + bc.ldloc(2); + bc.ret(); + + let mut code = bc.finish(); + patch_branch_target(&mut code, guard_ip, exit_ip); + let program = Program::new( + vec![ + Value::array(vec![Value::Int(97), Value::Int(98), Value::Int(99)]), + Value::bytes(vec![]), + Value::string(""), + Value::Int(0), + Value::Int(64), + Value::Int(1), + ], + code, + ) + .with_local_count(4); + let program = force_local_types( + program, + &[ + (0, ValueType::Array), + (1, ValueType::Bytes), + (2, ValueType::String), + (3, ValueType::Int), + ], + ); + let program = force_operand_types( + program, + &[ + (clt_ip as usize, (ValueType::Int, ValueType::Int)), + (from_ip as usize, (ValueType::Array, ValueType::Unknown)), + (utf8_ip as usize, (ValueType::Bytes, ValueType::Unknown)), + (add_ip as usize, (ValueType::Int, ValueType::Int)), + ], + ); + let mut vm = Vm::new(program); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::string("abc")]); + assert_native_ssa_specialized_trace( + &vm, + &vm.jit_snapshot(), + "ASCII bytes::to_utf8 loop", + &["bytes_from_array_u8", "bytes_to_utf8_ascii"], + ); +} + +#[test] +fn trace_jit_specializes_empty_array_construction_in_ssa() { + if !native_jit_supported() { + return; + } + let compiled = compile_source( + r#" + let mut i = 0; + let mut total = 0; + while i < 64 { + let values: [int] = []; + total = total + values.length; + i = i + 1; + } + total; + "#, + ) + .expect("array construction fixture should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(0)]); + assert_native_ssa_specialized_trace( + &vm, + &vm.jit_snapshot(), + "empty array construction loop", + &["array_new"], + ); +} + #[test] fn trace_jit_supports_shift_ops_in_ssa() { if !native_jit_supported() { @@ -4147,7 +5380,9 @@ fn trace_jit_supports_shift_ops_in_ssa() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "shift loop should not need native helper bridges, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -4258,7 +5493,9 @@ fn trace_jit_supports_eager_bool_ops_in_ssa() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "eager bool loop should not need native helper bridges, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -4319,7 +5556,9 @@ fn trace_jit_supports_float_comparisons_in_ssa() { let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "float compare loop should not use native helper bridges, bridge hits: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -4614,7 +5853,9 @@ fn trace_jit_specializes_literal_string_builtins_without_call_boundary() { ); let bridge_hits = vm.jit_native_bridge_stats_snapshot(); assert!( - bridge_hits.iter().all(|(_, count)| *count == 0), + bridge_hits + .iter() + .all(|(name, count)| *count == 0 || is_jit_state_boundary_bridge(name)), "literal string builtin loop should not use native helper bridges: {bridge_hits:?}\n{}", vm.dump_jit_info() ); @@ -4819,6 +6060,7 @@ fn trace_jit_executes_hot_loop_inside_script_callable_frame() { "#; let compiled = compile_source(source).expect("nested-frame loop should compile"); let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(false); vm.set_jit_config(JitConfig { enabled: true, hot_loop_threshold: 1, @@ -4833,8 +6075,195 @@ fn trace_jit_executes_hot_loop_inside_script_callable_frame() { assert!(vm.jit_native_exec_count() > 0); let snapshot = vm.jit_snapshot(); assert!( - snapshot.traces.iter().any(|trace| trace.frame_key == 0), - "expected a prototype-keyed trace: {}", + snapshot.traces.iter().any(|trace| { + trace.frame_key == 0 + && trace.terminal == JitTraceTerminal::LoopBack + && trace.executions > 0 + }), + "expected an executed prototype-keyed loopback trace: {}", + vm.dump_jit_info() + ); +} + +#[test] +fn trace_jit_region_cycles_without_external_handoffs() { + if !native_jit_supported() { + return; + } + let source = r#" + fn alternating_sum(limit: int) -> int { + let mut i = 0; + let mut total = 0; + while i < limit { + if i % 2 == 0 { + total = total + i; + } else { + total = total - i; + } + i = i + 1; + } + total + } + alternating_sum(4096); + "#; + let compiled = compile_source(source).expect("exit-heavy callable loop should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(false); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + + assert_eq!( + vm.run().expect("exit-heavy callable loop should run"), + VmStatus::Halted + ); + assert_eq!(vm.stack(), &[Value::Int(-2_048)]); + let first_native_execs = vm.jit_native_exec_count(); + let first_region_entries = vm.jit_native_region_entry_count(); + let first_internal_edges = vm.jit_native_internal_region_edge_count(); + let first_handoffs = vm.jit_native_link_handoff_count(); + let first_fallbacks = vm.jit_helper_fallback_count(); + assert_eq!(vm.jit_native_region_count(), 1, "{}", vm.dump_jit_info()); + assert!(first_native_execs > 0); + assert!(first_region_entries > 0, "{}", vm.dump_jit_info()); + assert!(first_internal_edges > 0, "{}", vm.dump_jit_info()); + + vm.reset_for_reuse(); + assert_eq!( + vm.run().expect("linked callable loop should run again"), + VmStatus::Halted + ); + assert_eq!(vm.stack(), &[Value::Int(-2_048)]); + let second_handoffs = vm + .jit_native_link_handoff_count() + .saturating_sub(first_handoffs); + assert!( + second_handoffs <= 2, + "callable cycle should remain in one native region, second_handoffs={second_handoffs}: {}", + vm.dump_jit_info() + ); + assert!(vm.jit_native_exec_count() > first_native_execs); + assert!(vm.jit_native_region_entry_count() > first_region_entries); + assert!(vm.jit_native_internal_region_edge_count() > first_internal_edges); + assert_eq!(vm.jit_helper_fallback_count(), first_fallbacks); +} + +#[test] +fn trace_jit_direct_links_cross_frame_call_and_return_edges() { + if !native_jit_supported() { + return; + } + let source = r#" + let mut i = 0; + let mut total = 0; + let delta = 3; + let add: fn(int) -> int = |value| value + delta; + while i < 4096 { + total = add(total); + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).expect("direct callable loop should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(true); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(12_288)]); + let first_direct = vm.jit_native_direct_link_count(); + let first_handoffs = vm.jit_native_link_handoff_count(); + + vm.reset_for_reuse(); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(12_288)]); + let direct_delta = vm.jit_native_direct_link_count() - first_direct; + let handoff_delta = vm.jit_native_link_handoff_count() - first_handoffs; + assert!(direct_delta > 12_000, "{}", vm.dump_jit_info()); + assert!(handoff_delta <= 3, "{}", vm.dump_jit_info()); + assert_eq!(vm.jit_native_region_count(), 0); +} + +#[test] +fn trace_jit_missing_dynamic_return_target_never_uses_stale_static_continuation() { + if !native_jit_supported() { + return; + } + let source = r#" + let add: fn(int) -> int = |value| value + 1; + let mut i = 0; + let mut total = 0; + while i < 256 { + total = add(total); + total = total + 10; + i = i + 1; + } + total = add(total); + total = total + 100; + total; + "#; + let compiled = compile_source(source).expect("multiple continuation sites should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(true); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(2_917)]); + + vm.reset_for_reuse(); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(2_917)]); +} + +#[test] +fn trace_jit_direct_link_slots_clear_and_republish_after_mode_toggle() { + if !native_jit_supported() { + return; + } + let source = r#" + let mut i = 0; + let mut total = 0; + while i < 512 { + if i % 2 == 0 { total = total + 3; } else { total = total + 5; } + i = i + 1; + } + total; + "#; + let compiled = compile_source(source).unwrap(); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(true); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 512, + }); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(2_048)]); + assert!(vm.jit_native_direct_link_count() > 500); + + vm.set_jit_native_direct_links_enabled(false); + vm.reset_for_reuse(); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(2_048)]); + assert_eq!(vm.jit_native_direct_link_count(), 0); + + vm.set_jit_native_direct_links_enabled(true); + vm.reset_for_reuse(); + assert_eq!(vm.run().unwrap(), VmStatus::Halted); + assert_eq!(vm.stack(), &[Value::Int(2_048)]); + assert!( + vm.jit_native_direct_link_count() > 500, + "{}", vm.dump_jit_info() ); } @@ -4991,6 +6420,37 @@ fn trace_jit_call_value_yields_and_resumes_host_callable_without_losing_frame_st assert!(vm.dump_jit_info().contains("interpreter fallbacks: 0")); } +#[test] +fn trace_jit_missing_dynamic_return_target_does_not_use_stale_static_slot() { + if !native_jit_supported() { + return; + } + let source = r#" + fn inc(x: int) -> int { x + 1 } + let mut i = 0; + let mut value = 0; + while i < 32 { + value = inc(value); + i = i + 1; + } + value = inc(value) + 100; + value; + "#; + let compiled = compile_source(source).expect("stale return target fixture should compile"); + let mut vm = Vm::new(compiled.program.with_local_count(compiled.locals)); + vm.set_jit_native_direct_links_enabled(true); + vm.set_jit_config(JitConfig { + enabled: true, + hot_loop_threshold: 1, + max_trace_len: 256, + }); + assert_eq!( + vm.run().expect("stale return target fixture should run"), + VmStatus::Halted + ); + assert_eq!(vm.stack(), &[Value::Int(133)]); +} + #[test] fn trace_jit_links_nested_dynamic_script_callables_without_interpreter_handoff() { if !native_jit_supported() { diff --git a/tests/jit/perf_tests.rs b/tests/jit/perf_tests.rs index 63d7cfe2..3b2e30ba 100644 --- a/tests/jit/perf_tests.rs +++ b/tests/jit/perf_tests.rs @@ -1447,7 +1447,7 @@ fn build_host_call_latency_source(name: &str, arity: u8, calls: i64) -> String { match arity { 0 => format!( r#" - fn {name}(); + fn {name}() -> int; let mut i = 0; let mut sum = 0; while i < {calls} {{ @@ -1459,7 +1459,7 @@ fn build_host_call_latency_source(name: &str, arity: u8, calls: i64) -> String { ), 1 => format!( r#" - fn {name}(x); + fn {name}(x) -> int; let mut i = 0; let mut sum = 0; while i < {calls} {{ diff --git a/tests/vm/vm_runtime_tests.rs b/tests/vm/vm_runtime_tests.rs index 9a838f95..f9c6fcb4 100644 --- a/tests/vm/vm_runtime_tests.rs +++ b/tests/vm/vm_runtime_tests.rs @@ -19,6 +19,23 @@ fn non_yielding_pends(_: &[Value]) -> Result { Ok(CallOutcome::Pending(7)) } +fn non_yielding_returns_bool(_: &[Value]) -> Result { + Ok(CallOutcome::Return(vm::CallReturn::one(Value::Bool(true)))) +} + +#[test] +fn non_yielding_args_return_type_contract_is_enforced_before_jit_compilation() { + let compiled = + compile_source("fn action() -> int; action();").expect("host call source should compile"); + let mut vm = Vm::new(compiled.program); + vm.bind_static_non_yielding_args_function("action", non_yielding_returns_bool); + + assert!(matches!( + vm.run().expect_err("return type mismatch should fail"), + vm::VmError::TypeMismatch("int") + )); +} + #[test] fn non_yielding_args_contract_is_enforced_before_jit_compilation() { let cases: [(&str, StaticHostArgsFunction); 4] = [