Arm64: Improve support for HW_Flag_ReturnsPerElementMask

[snickolls-arm]

When wrapping an intrinsic node that has an embedded mask with a ConditionalSelect, ensure that the constant node in op3 has a mask type when the intrinsic has the HW_Flag_ReturnsPerElementMask flag.

Build out further support for ConditionalSelect_Predicates, and use this to wrap nodes with HW_Flag_ReturnsPerElementMask. Add GenTree::IsSelectZero and update various areas in HW intrinsic codegen to ensure this intrinsic assembles correctly.

[dotnet-policy-service]

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch
See info in area-owners.md if you want to be subscribed.

[dhartglassMSFT]

FYI if it's not known, assert in PR testing

11:15:45.200 Running test: JIT/opt/SVE/ConditionalSelectConstants/ConditionalSelectConstants.cmd

Assert failure(PID 5320 [0x000014c8], Thread: 6404 [0x1904]): Assertion failed 'maskBit == 0' in 'ConditionalSelectConstants:ConditionalSelect1CC():System.Numerics.Vector`1[int]' during 'Morph - Global' (IL size 24; hash 0x36e039d8; FullOpts)

    File: D:\a\_work\1\s\src\coreclr\jit\simd.h:2145
    Image: C:\h\w\AEC00944\p\corerun.exe


Return code:      1

[snickolls-arm]

FYI if it's not known, assert in PR testing

11:15:45.200 Running test: JIT/opt/SVE/ConditionalSelectConstants/ConditionalSelectConstants.cmd

Assert failure(PID 5320 [0x000014c8], Thread: 6404 [0x1904]): Assertion failed 'maskBit == 0' in 'ConditionalSelectConstants:ConditionalSelect1CC():System.Numerics.Vector`1[int]' during 'Morph - Global' (IL size 24; hash 0x36e039d8; FullOpts)

    File: D:\a\_work\1\s\src\coreclr\jit\simd.h:2145
    Image: C:\h\w\AEC00944\p\corerun.exe


Return code:      1

I didn't see thanks, will take a look.

[Copilot]

Pull request overview

This PR extends Arm64 SVE support for intrinsics that return per-element masks by introducing shared recognition of SVE conditional-select variants, adding a “select-zero” helper, updating containment/codegen to treat mask-zero similarly to vector-zero, and updating constant-folding / emitter behavior to match SVE predicate element semantics.

Changes:

• Adds HWIntrinsicInfo::IsSveConditionalSelect and GenTree::IsSelectZero, and updates lowering/LSRA/codegen paths to use them.
• Updates SVE mask creation intrinsics to be treated as returning per-element masks and adjusts import/constant construction accordingly.
• Adjusts SVE emitter options handling and updates/extends a JIT test to validate the new mask element representation.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/tests/JIT/opt/SVE/ConditionalSelectConstants.cs	Updates expectations and adds a smoke check validating mask element values.
src/coreclr/jit/simd.h	Changes mask-to-vector constant evaluation semantics on Arm64 and adds helper to convert elementwise mask values to bitwise masks for folding.
src/coreclr/jit/lsraarm64.cpp	Treats both vector and predicate conditional-select forms uniformly when identifying embedded/contained selects.
src/coreclr/jit/lowerarmarch.cpp	Expands lowering/containment to include predicate conditional-select and uses IsSelectZero for zero-detection.
src/coreclr/jit/hwintrinsiclistarm64sve.h	Marks SVE CreateTrue/FalseMask intrinsics as returning per-element masks; updates containment flag for predicate select.
src/coreclr/jit/hwintrinsiccodegenarm64.cpp	Switches embedded-mask setup and containment checks from vector-zero to select-zero, enabling mask-zero handling.
src/coreclr/jit/hwintrinsicarm64.cpp	Imports SVE false masks via gtNewZeroConNode and updates true-mask constant folding to use mask constants.
src/coreclr/jit/hwintrinsic.h	Introduces IsSveConditionalSelect helper for consistent conditional-select recognition.
src/coreclr/jit/gentree.h	Adds GenTree::IsSelectZero helper for conditional-select zero detection across vectors/masks.
src/coreclr/jit/gentree.cpp	Extends gtNewZeroConNode to create mask-zero constants and updates folding for SVE mask representation.
src/coreclr/jit/emitarm64sve.cpp	Forces byte arrangement for predicate encodings where required and relaxes/adjusts scalable option assertions.

Comments suppressed due to low confidence (1)

src/coreclr/jit/lowerarmarch.cpp:3636

• TryContainingCselOp now accepts both NI_Sve_ConditionalSelect and NI_Sve_ConditionalSelect_Predicates, but the containment fast-path is still gated on childNode->Op(3)->IsVectorZero(). For the predicate variant, op3 is expected to be a mask-zero (pfalse) rather than a vector-zero, so this will never trigger and makes the new broader check ineffective for mask selects.

    if (!HWIntrinsicInfo::IsSveConditionalSelect(childNode->GetHWIntrinsicId()))
    {
        return false;
    }

    if (childNode->Op(2)->IsEmbMaskOp())
    {
        // Do not optimise if the conditional select node is added to embed the mask for its op2. Such conditional
        // select nodes are optimised out while emitting the embedded variant of an instruction associated with op2.
        assert(childNode->Op(2)->isContained());
        return false;
    }

    bool canContain = false;

    var_types simdBaseType = parentNode->GetSimdBaseType();
    if (childNode->Op(3)->IsVectorZero())
    {

[tannergooding]

For now I'd rather just get this PR completed so I can start contributing the Vector intrinsic implementations I've done using SVE.

@snickolls-arm, as an FYI this is going to change a bit with #130049.

It is effectively merging most of the tables for the xplat intrinsics into a new combined table and the import logic itself into a combined handler as well. The handling between Arm64 AdvSimd and xarch was already nearly identical and the same will again be true for Arm64 SVE and the WASM support we're onboarding, with most of the differences (if any) being in the gtNewSimd* helper APIs or in the actual codegen handling, which was often special.

That should make it a bit simpler to lightup for SVE, as you'll not need to extend or duplicate the tables again or add a bunch of NI_VectorT_* entries alongside the existing NI_Vector64_* and NI_Vector128_* handlers. Instead, it's purely NI_Vector_* for all cases and differentiated based on simdSize (i.e. SIZE_UNKNOWN for SVE).

[snickolls-arm]

For now I'd rather just get this PR completed so I can start contributing the Vector intrinsic implementations I've done using SVE.

@snickolls-arm, as an FYI this is going to change a bit with #130049.

It is effectively merging most of the tables for the xplat intrinsics into a new combined table and the import logic itself into a combined handler as well. The handling between Arm64 AdvSimd and xarch was already nearly identical and the same will again be true for Arm64 SVE and the WASM support we're onboarding, with most of the differences (if any) being in the gtNewSimd* helper APIs or in the actual codegen handling, which was often special.

That should make it a bit simpler to lightup for SVE, as you'll not need to extend or duplicate the tables again or add a bunch of NI_VectorT_* entries alongside the existing NI_Vector64_* and NI_Vector128_* handlers. Instead, it's purely NI_Vector_* for all cases and differentiated based on simdSize (i.e. SIZE_UNKNOWN for SVE).

Thanks for the heads up. This should reduce the diffs I have for implementing the Vector<T> API quite a bit, I'll hold off from sending too much of that then until #130049 is merged and I've rebased on it.

[tannergooding]

LGTM, but I'd really like for us to fix the SVE usages of ReturnsPerElementMask to be consistent with its usage on other targets/platforms.

CC. @dotnet/jit-contrib, @dhartglassMSFT for secondary review

[dhartglassMSFT]

I'm catching up here still, so apologies if I'm making you repeat yourself.

Is the idea the JIT will now maintain two alternate representations for a mask, ex for short: 0XFFFF...F vs. 0b0101...01 ?

[snickolls-arm]

Is the idea the JIT will now maintain two alternate representations for a mask, ex for short: 0XFFFF...F vs. 0b0101...01 ?

Short term yes, but long term we want to have the former representation so it's consistent with other architectures. Then the optimization code doesn't have to diverge for SVE like I've done here. I've noted this on #120599 to schedule in soon.

I just didn't want to bloat the PR too much, as it distracts from what I was trying to do to support ConditionalSelect on predicates. It'll cause a bit of churn in test code because the latter representation has been baked into a lot of that.

[github-actions]

Workflow state for the Holistic Review Orchestrator.

{
  "version": 5,
  "last_dispatched_commit": "bba2c056585e9a61b3e275825eb4f514c6e2b072",
  "last_dispatched_base_ref": "main",
  "last_dispatched_base_sha": "bbdd508a9371af9922e65ad073f9e372bc5c208e",
  "last_reviewed_commit": "bba2c056585e9a61b3e275825eb4f514c6e2b072",
  "last_reviewed_base_ref": "main",
  "last_reviewed_base_sha": "bbdd508a9371af9922e65ad073f9e372bc5c208e",
  "last_recorded_worker_run_id": "29684898666",
  "review_attempt_commit": "",
  "review_attempt_base_ref": "",
  "review_attempt_count": 0,
  "max_review_attempts": 5,
  "review_history_format": "holistic-review-disclosure-v1",
  "review_history": [
    {
      "commit": "bba2c056585e9a61b3e275825eb4f514c6e2b072",
      "review_id": 4730673113
    }
  ]
}

[github-actions]

Holistic Review

Motivation: The problem is real and well-motivated. When wrapping an intrinsic with HW_Flag_ReturnsPerElementMask in a ConditionalSelect, the false-value constant (op3) previously used a vector-zero even for mask-typed results, and the mask Create*Mask intrinsics were not marked as returning per-element masks. This PR makes the type/flag invariant consistent (HW_Flag_ReturnsPerElementMask <==> TYP_MASK) and builds out ConditionalSelect_Predicates so mask-producing operations assemble correctly.

Approach: The approach is sound and matches maintainer direction (jakobbotsch requested exactly this consistency work). Introducing IsSveConditionalSelect/IsZeroForSelect helpers, treating mask-zero like vector-zero across lowering/LSRA/codegen, and forcing byte arrangement for the single predicate-select encoding are all reasonable. The main risk is a deliberate change to Arm64 mask->vector constant-folding semantics that must be applied consistently across all consumers.

Summary: ⚠️ Needs Human Review. The core direction is correct and endorsed by the JIT team, but changing EvaluateSimdCvtMaskToVector to emit {0, 1} instead of {0, AllBitsSet} on Arm64 is a cross-cutting semantic change. The author compensated in gtFoldExprHWIntrinsic via the new ConvertToBitWiseMask, but the parallel constant-folding reconstruction in valuenum.cpp appears to consume the same helper without that conversion. A JIT reviewer should confirm whether that VN path is also affected (it lies in unchanged code, so it is out of this review's strict change-scope but is directly coupled to a changed helper). See the inline comment on simd.h.

---

Detailed Findings

⚠️ Correctness — Arm64 mask->vector semantics change may leave a stale value-numbering consumer

EvaluateSimdCvtMaskToVector (and EvaluateSimdPatternToVector) now produce element value 1 rather than AllBitsSet on Arm64. The gtFoldExprHWIntrinsic ConditionalSelect folding path was updated to call ConvertToBitWiseMask before feeding the value into EvaluateBinarySimd(GT_AND, ...). However, valuenum.cpp (~line 9645, NI_Sve_ConditionalSelect branch under TARGET_ARM64) performs the same (y & x) | (z & ~x) reconstruction using EvaluateSimdCvtMaskToVector's result directly, without the new conversion. If reachable with constant operands, this would fold to an incorrect result (masking with 1 instead of AllBitsSet). This is coupled to a changed helper but lives in a file not in this PR's diff — flagged for human confirmation rather than as a strict in-scope blocker. Details in the inline comment on simd.h.

✅ Invariant enforcement — HW_Flag_ReturnsPerElementMask <==> TYP_MASK

The new assert in LowerHWIntrinsic correctly codifies the intended invariant and is placed early enough to catch inconsistencies before embedded-mask analysis depends on it. The hwintrinsiclistarm64sve.h flag additions on the CreateTrue/FalseMask* intrinsics are consistent with importing them as TYP_MASK constants via gtNewMskConNode/gtNewZeroConNode.

✅ Emitter byte-arrangement handling

Forcing INS_OPTS_SCALABLE_B for IF_SVE_DC_3A and the predicate sel (IF_SVE_CZ_4A) encodings is justified by the accompanying comment: these are bitwise predicate operations with a single encoding, so lane arrangement is irrelevant to the result. Relaxing the assert to insOptsScalable(opt) before normalizing to byte is reasonable.

✅ Helper consolidation

IsSveConditionalSelect and IsZeroForSelect are applied consistently across hwintrinsiccodegenarm64.cpp, lowerarmarch.cpp, and lsraarm64.cpp, replacing the previous == NI_Sve_ConditionalSelect / IsVectorZero() checks. This correctly extends the existing vector-select handling to the predicate variant.

⚠️ Test coverage — predicate-select paths only lightly exercised

The test update (ConditionalSelectConstants.cs) adds a useful Smoke() check and corrects expected values that shift because mask elements are now 1 rather than sign-extended AllBitsSet (e.g. -3 -> 3, 4080 -> 16). This validates the changed CreateTrueMask constant representation. However, there is no test that directly exercises ConditionalSelect_Predicates (mask-typed op3 = CreateFalseMask) end-to-end, nor the value-numbering constant-fold path noted above. Given this is an observable behavior change with a clear repro area, a targeted test for the predicate-select and the folded-VN path would strengthen confidence. Note the earlier PR-testing assert (maskBit == 0 firing in ConvertToBitWiseMask) was reported and addressed during development; confirm current CI is green before merge.

Note

This review was generated by this repository's Holistic Review agentic workflow to complement the built-in Copilot review.

Generated by Holistic Review · 159.6 AIC · ⌖ 11.5 AIC · ⊞ 10K

[github-actions]

This changes the Arm64 semantics of EvaluateSimdCvtMaskToVector from producing {0, AllBitsSet} elements to {0, 1} elements. The gtFoldExprHWIntrinsic consumer was updated to compensate via the new ConvertToBitWiseMask, but the near-identical constant-folding reconstruction in valuenum.cpp (~line 9645, the NI_Sve_ConditionalSelect branch under TARGET_ARM64) uses the same EvaluateSimdCvtMaskToVector result directly in GT_AND/GT_AND_NOT/GT_OR to rebuild (y & x) | (z & ~x). With mask elements now 1 instead of AllBitsSet, that VN path will compute an incorrect folded ConditionalSelect for constant inputs (e.g. 1 & value instead of AllBitsSet & value). Please confirm whether this VN branch also needs the ConvertToBitWiseMask conversion, or add a test that exercises constant-folded SVE ConditionalSelect through value numbering. The lowerarmarch.cpp caller at line ~1024 uses byte base type where element==1 still round-trips correctly through the subsequent vector->mask conversion, so it appears unaffected, but the VN reconstruction does not go through that round-trip.