JIT: Reuse enregistered FP/SIMD/mask constants across non-last uses#130312
JIT: Reuse enregistered FP/SIMD/mask constants across non-last uses#130312tannergooding wants to merge 13 commits into
Conversation
LSRA previously could only reuse an already-enregistered constant when the earlier occurrence was at its last use, forcing redundant rematerialization (e.g. repeated `xorps`) of identical CNS_DBL/CNS_VEC/CNS_MSK values. At build time, coalesce overlapping identical FP/SIMD/mask constants (still pending in the defList) into a single interval, adding later occurrences as redefinitions. At allocation time, elide a redundant redef via SetReuseRegVal when the register already holds the value. Clear the prior use's lastUse so the coalesced interval remains one continuous range, and never restore a constant previousInterval whose register was borrowed and overwritten (it must be rematerialized instead). These constants have no GC liveness considerations, so coalescing is safe. Fixes dotnet#70182 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
Pull request overview
This PR enhances CoreCLR JIT’s LSRA handling of floating-point/SIMD/mask constants so identical constants that are simultaneously live can share a single interval/register, reducing redundant rematerialization. It also adds a targeted JIT regression test covering several representative patterns.
Changes:
- Coalesce overlapping identical
GT_CNS_DBL/GT_CNS_VEC/GT_CNS_MSKdefinitions into a shared constant interval during LSRA build. - During allocation, mark coalesced constant redefinitions as
ReuseRegValwhen the register already contains the constant; also prevent restoring apreviousIntervalwhen it is a constant. - Add an xUnit regression test (
Runtime_70182) and wire it intoRegression_o_1.csproj.
Show a summary per file
| File | Description |
|---|---|
| src/coreclr/jit/lsrabuild.cpp | Adds constant-node equality + build-time coalescing; updates lastUse behavior for constant intervals. |
| src/coreclr/jit/lsra.cpp | Prevents restoring constant previousInterval; redefinition reuse when constant already materialized. |
| src/coreclr/jit/lsra.h | Declares new LSRA build helpers for constant reuse/coalescing. |
| src/tests/JIT/Regression/Regression_o_1.csproj | Includes the new regression test source file. |
| src/tests/JIT/Regression/JitBlue/Runtime_70182/Runtime_70182.cs | New xUnit regression coverage for overlapping identical constants (double/SIMD/mask scenarios). |
Copilot's findings
- Files reviewed: 5/5 changed files
- Comments generated: 2
Address review feedback: getConstantIntervalForReuse can coalesce a later constant definition (RefTypeDef) into an interval after an earlier occurrence has already been consumed (RefTypeUse). The prior lastUse-clearing only ran in the RefTypeUse branch, leaving that earlier use marked as lastUse and allowing the coalesced interval to be freed mid-range. Move the clearing so it runs for every reference added to a constant interval. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Fixes an ARM32 LSRA assert (lsra.cpp:6121) in cases such as System.Numerics.Vector3:get_Zero, where FEATURE_SIMD is off and 'return default(Vector3)' lowers to RETURN(FIELD_LIST(CNS_DBL 0, CNS_DBL 0, CNS_DBL 0)). Each FIELD_LIST operand is used at the same LSRA location (the RETURN) but requires a distinct fixed ABI return register. Coalescing the identical constants into a single interval forced that interval to occupy multiple registers at once. getConstantIntervalForReuse now skips coalescing when the new constant and any already-coalesced pending definition share a consuming node (resolved through contained users via getConsumingNode), since that is exactly when two uses land at the same location. Constants feeding different consumers (the intended reuse case) are unaffected. Also wires up the previously-unused canHandle early-out. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The constant-reuse coalescing has no effect in MinOpts (enregisterLocalVars disabled), and running the defList scan there added measurable throughput cost (up to +0.34%). Gate the whole helper on OptimizationEnabled to match the existing isMatchingConstant reuse path and avoid that overhead. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The previous test defined its vector constant once in a local (a single GT_CNS_VEC) and consumed each scalar literal immediately, so it never drove getConstantIntervalForReuse. Reshape it around System.Numerics shapes that actually keep identical FP/SIMD constants live across distinct consumers (Vector3/Vector4.Normalize, Quaternion.Lerp, Matrix4x4.Decompose), running each at full opts behind NoInlining wrappers with runtime inputs and validating against an independent scalar reference within tolerance. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A coalesced FP/SIMD/mask constant interval can have several uses that read the value directly from a single spill temp. updateMaxSpill was designed for single-use tree temps and decremented the live spill count at every such use, underflowing currentSpill (assert 'currentSpill[type] > 0' at lsra.cpp:7822) on x86 where register pressure causes the constant to be spilled and used from memory more than once. Only release the temp on the last use; earlier uses keep it reserved. No-op for single-use intervals. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Only floating-point/SIMD/mask constants are ever coalesced (areSameConstantNodes), so integer constant intervals are never multi-ref. Gate the lastUse-clearing on a non-integer register type to avoid the (harmless but pointless) work on every integer constant reference. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Coalescing identical enregistered FP/SIMD/mask constants into a single multi-reference interval is unsafe when one of the shared uses is a read-modify-write operand (e.g. an FMA addend or a non-VEX binary op on xarch). The RMW reuses the operand's register for its result, clobbering the constant at the point of use while later references of the coalesced interval still expect the value. Because a coalesced constant is spillable, under register pressure the allocator may store-spill the already-clobbered register and reload garbage for the later uses, producing wrong results (observed under JitStressRegs=0x2/0x3/0x7). Decline coalescing whenever a candidate constant's consumer is such an RMW operation. This is intentionally conservative (it does not distinguish which operand is reused) and can be relaxed once spilled FP/SIMD/mask constants are rematerialized instead of stored and reloaded. Also fix spill-temp accounting in updateMaxSpill: a coalesced constant can be reloaded and immediately re-spilled at the same refposition, which leaves the concurrent spill count unchanged, so the reload must not decrement the count in that case. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
A constant use that is not assigned a register reads its value directly from the read-only data section, not from a spill temp, so it does not free a spill temp. The RegOptional/memory branch in updateMaxSpill was still decrementing currentSpill for such uses, which could drive the count below zero (asserting 'currentSpill[type] > 0') and, worse, under-count maxSpill so codegen under-allocated spill temps. Skip the decrement for constant intervals: a constant def is only store-spilled when a later use needs the value in a register (a reload), and that temp is released on the reload instead. Also assert-guard the reload branch consistently. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
This basically has no diffs until after the rematerialization pr (130313) goes in, which will allow some of the pessimization to be avoided. |
|
LSRA changes. @jakobbotsch, PTAL. |
|
LGTM. [ACK] [ACK] [ACK] [ACK]
|
|
This was essentially trying to resolve the issue by explicitly identifying and merging intervals where applicable, but its tricky and because its LSRA almost any change even just limiting it to optimized code only has high TP cost. The new pass for constant reuse that Arm is attempting to add in #128844 may actually be a better overall approach and then we could potentially remove this special constant reuse tracking in LSRA |
|
Workflow state for the Holistic Review Orchestrator. {
"version": 5,
"last_dispatched_commit": "ef52d3466ca9e7d9b9bebe52f312fc13a7dd0344",
"last_dispatched_base_ref": "main",
"last_dispatched_base_sha": "f00ccfddcd92f805d2d721ccaabfad6a9089e845",
"last_reviewed_commit": "ef52d3466ca9e7d9b9bebe52f312fc13a7dd0344",
"last_reviewed_base_ref": "main",
"last_reviewed_base_sha": "f00ccfddcd92f805d2d721ccaabfad6a9089e845",
"last_recorded_worker_run_id": "29680708116",
"review_attempt_commit": "",
"review_attempt_base_ref": "",
"review_attempt_count": 0,
"max_review_attempts": 5,
"review_history_format": "holistic-review-disclosure-v1",
"review_history": [
{
"commit": "ef52d3466ca9e7d9b9bebe52f312fc13a7dd0344",
"review_id": 4730566878
}
]
} |
There was a problem hiding this comment.
Holistic Review
Motivation: Justified and backed by a concrete issue (#70182): LSRA could only reuse an already-enregistered FP/SIMD/mask constant when the prior occurrence was its last use, forcing repeated rematerialization (e.g. several xorps for Zero) when 4-5 registers get initialized to the same value in code like Matrix4x4.Decompose. This is a real, measurable code-quality gap.
Approach: Sound and appropriately conservative. At build time, identical FP/SIMD/mask constants still pending in defList are coalesced into one interval (later occurrences become redefinitions), with several safety guards: integer constants are excluded, same-consumer constants are rejected (a single interval can't satisfy two fixed-register uses at one location), read-modify-write consumers are declined (the shared reg would be clobbered at point of use), and partial-callee-save types are skipped. At allocation time a redundant redef is elided via SetReuseRegVal, lastUse on the coalesced use is cleared to keep the interval one continuous range, and a borrowed/overwritten constant previousInterval is no longer restored. The spill-accounting assert relaxations are correctly reasoned and documented.
Summary: ✅ LGTM. The change is well-scoped, the correctness reasoning for each guard and each relaxed spill assert is documented inline, GC liveness is correctly noted as a non-concern for these constant kinds, and it carries a meaningful regression test exercising the real vector-math shapes that drive the path. Already approved by area maintainer @JulieLeeMSFT. My independent read found no blocking issues; the notes below are confirmations and one observation for the author/human reviewer to confirm.
Detailed Findings
✅ Coalescing safety guards — conservative and well-reasoned
getConstantIntervalForReuse rejects coalescing whenever any matching pending def shares the consumer, has an undeterminable consumer, or feeds a read-modify-write consumer, and skips partial-callee-save types. The same-consumer rejection (a single interval cannot occupy multiple fixed registers at one location, e.g. FIELD_LIST operands) and the RMW rejection (shared reg clobbered at point of use while later refs still expect the value, unsafe because these constants are store-spilled rather than rematerialized) are both correct. The lazy computation of treeConsumer is a reasonable micro-optimization given most defs won't match.
✅ Spill accounting relaxations — correctly bounded
The two updateMaxSpill assert relaxations (reload && spillAfter for a coalesced constant, and a constant use with no assigned register reading from the read-only data section) are each gated on interval->isConstant, so non-constant intervals retain the original strict assert(currentSpill[type] > 0); currentSpill[type]--; behavior. The reasoning that a constant use reads from rodata (not a spill temp) and that a coalesced reload+respill leaves the concurrent spill count unchanged is consistent with the coalescing model.
✅ Test quality — targeted and self-validating
Runtime_70182.cs drives the exact shapes that exercise the path (Vector3/Vector4 Normalize, Quaternion.Lerp, Matrix4x4.Decompose), keeps inputs in non-readonly statics so the JIT can't constant-fold, and validates against an independent scalar reference with a tolerance — so a bad register reuse (which yields garbage, not a small rounding delta) is caught. Correctly placed in JitBlue and registered in Regression_o_1.csproj. Because AVX-512 masking makes the constants register-resident, note that coverage of the register-resident path depends on the CI hardware exercising it — which the broad runtime test matrix should provide.
💡 canRestorePreviousInterval now excludes all constant intervals — confirm intent
The new guard returns false for any previousInterval->isConstant, not only the coalesced FP/SIMD/mask kind. Integer constant intervals are never coalesced, so this broadens the restore-suppression to a class of intervals that were previously restorable. This is very likely intentional (a borrowed constant register was overwritten and must be rematerialized regardless of constant kind, which is always cheap for the constants LSRA marks isConstant), and it looks correct, but it is worth an explicit confirmation from the author that suppressing restore for integer constants here has no throughput/CQ downside in the non-coalesced path.
Note
This review was generated by this repository's Holistic Review agentic workflow to complement the built-in Copilot review.
Generated by Holistic Review · 142.7 AIC · ⌖ 10.7 AIC · ⊞ 10K
LSRA previously could only reuse an already-enregistered constant when the earlier occurrence was at its last use, forcing redundant rematerialization (e.g. repeated
xorps) of identicalCNS_DBL/CNS_VEC/CNS_MSKvalues.Change
defList) into a single interval, adding later occurrences as redefinitions.SetReuseRegValwhen the register already holds the value. Clear the prior use''slastUseso the coalesced interval remains one continuous range, and never restore a constantpreviousIntervalwhose register was borrowed and overwritten (it must be rematerialized instead).These constants have no GC liveness considerations, so coalescing is safe.
Fixes #70182
Note
This pull request was authored with the assistance of GitHub Copilot.