[Perf] Linux/arm64: 1 Regression on 7/8/2026 8:49:11 AM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	arm64
OS	azurelinux 3.0
Queue	CobaltAzureLinux
Baseline	c76054f5cb1aae6ab146ec7a8db70ca3aeda4ac9
Compare	3ade0b55c797a31f6800040e3ae3c5c75bc0d2dc
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tests.Perf_Vector2

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
NormalizeBenchmark - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	4.11 ns	5.75 ns	1.40	0.22	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
py .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tests.Perf_Vector2*'

Details

System.Numerics.Tests.Perf_Vector2.NormalizeBenchmark

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Verified regression in NormalizeBenchmark — caused by commit f21f194 ("Refactor the various System.Numerics Vector/Matrix/Quaternion/Plane types to req").

Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)

Likely Cause: f21f1945632f432952967797828737587db11951

🤖 Proposed Next Actions: Known-cause regression — transfer & notify — 0 done, 4 dry-run (rule R4_known_regression_new — see Full Analysis for the table). · 🦆 Re-analysis: ✅ concurs

📊 Full Analysis (click to expand)

Summary: Clear regression in System.Numerics.Tests.Perf_Vector2.NormalizeBenchmark (+39.8%) attributed to commit f21f194 which refactored System.Numerics Vector types to require less inlining. Finding confidence 4/5 (strong historical data, 300 runs, z=3.89), regression confidence 5/5 (surgical change directly to Vector2.Normalize code path, extreme z-score, stable step change on arm64).

Issue Overview

• Issue: [Perf] Linux/arm64: 1 Regression on 7/8/2026 8:49:11 AM +00:00 #130419 - [Perf] Linux/arm64: 1 Regression on 7/8/2026 8:49:11 AM +00:00
• Date: 2026-07-08 | Queue: AzureLinux.3.Cobalt.Arm64.Perf | OS/Arch: Linux/arm64
• Commits: c76054f5cb → 3ade0b55c7 (18 commits)
• Labels: untriaged, perf-regression, os-linux, branch-refs/heads/main, arch-arm64, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
• Historical Data: Available (300 runs) - high statistical confidence
• Runtime Stack: Desktop + CoreCLR (implicit)

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
System.Numerics.Tests.Perf_Vector2.NormalizeBenchmark	4.11 ns	5.75 ns	+39.83%	4.21	0.395	300	3.891	18.76%	N	Step change, high confidence

Legend: μ=mean, σ=stddev, n=runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 - Strong historical data (300 prior runs), clear step change pattern with all 6 post-regression runs clustering tightly around 5.7 ns. CV is 9.4% due to occasional low outliers in the before-series, but the regression signal is unambiguous.

Candidate Stack Relevance

Commit	Touched Paths (summary)	Relevance	Notes
f21f1945632f	src/libraries/.../System/Numerics/Vector2.cs, Vector3.cs, Vector4.cs, Matrix3x2.cs, Matrix4x4.cs, Quaternion.cs, Plane.cs, Vector128.Numerics.cs	Relevant	Direct modification to Vector2.Normalize code path; refactored inlining strategy
fdecb6b73237	src/coreclr/vm/excep.cpp, exceptionhandling.cpp, frames.h	Relevant	CoreCLR exception frame fix; unlikely to affect Vector2 perf
d93f5cd78d48	src/coreclr/jit/emit.cpp	Relevant	JIT emit change for prolog/epilog splitting; could indirectly affect codegen

Confirmed Regressions

System.Numerics.Tests.Perf_Vector2.NormalizeBenchmark (+39.83%)

• Delta: 4.11 ns → 5.75 ns (+39.83%)
• Historical Context: μ=4.21 ns (σ=0.395, n=300); z-score=3.891; adaptive threshold=18.76%
• Assessment: Step change. All 6 post-regression runs cluster tightly around 5.7 ns, a clear discontinuity from the pre-regression mean of ~4.21 ns. The delta of +39.83% is more than double the adaptive threshold of 18.76%.
• Finding Confidence: 4/5 - Comprehensive historical data (300 runs), unambiguous step change pattern, low clustering variance in post-regression runs.
• Regression Confidence: 5/5 - Surgical change directly to Vector2 code, extreme outlier (z=3.89 >> 3.0 threshold), 18-commit range with one overwhelmingly likely candidate, stable history makes noise virtually impossible.
• Likely Cause: f21f1945632f - "Refactor the various System.Numerics Vector/Matrix/Quaternion/Plane types to require less inlining (Refactor the various System.Numerics Vector/Matrix/Quaternion/Plane types to require less inlining #130274)"
  • Affected Areas: BCL / System.Numerics / Vector intrinsics / Inlining
  • Stack Relevance: Relevant
  • Rationale: This commit directly modified Vector2.cs (158 additions, 99 deletions) as part of a large refactor to reduce inlining depth in System.Numerics types. The commit explicitly states it makes functions "less reliant on the inliner" and changes which functions get inlined. On arm64 (Cobalt), the reduced inlining for Vector2.Normalize likely prevents the JIT from fully optimizing the hot path, resulting in the +39.8% regression. The benchmark directly calls Vector2.Normalize(VectorTests.Vector2Value) which is the exact code path modified.
  • Candidates Considered:
    • f21f1945632f - Direct modification to Vector2.cs Normalize implementation and inlining attributes
    • d93f5cd78d48 - JIT emit prolog/epilog splitting change; unlikely to affect such a small method
    • fdecb6b73237 - Exception frame fix; unrelated to Vector2 hot path
• Verification Status: Not verified - high confidence from code analysis alone given the surgical match

Open Questions

• Historical Data: Available and strong (300 runs, sufficient post-regression data)
• Confidence Limitations: None significant - the commit is a direct hit on the exact code path
• Verification Needed: Consider verifying with RunWithBuildAtHash at commits before/after f21f194 to definitively confirm, though confidence is already high
• Bisection Status: Not performed - single commit attribution with high confidence makes bisection unnecessary
• ARM64-Specific Impact: The refactoring may have different performance characteristics on arm64 vs x64 due to differences in how the JIT handles inlining decisions and SIMD/vector codegen on each architecture. Worth checking if x64 is also affected.

Recommended Actions

1. Notify author - Alert tannergooding about the arm64 regression in Vector2.Normalize (+39.8%) from the inlining refactor PR Refactor the various System.Numerics Vector/Matrix/Quaternion/Plane types to require less inlining #130274
2. ARM64 codegen review - Review the generated arm64 JIT assembly for Vector2.Normalize before and after the refactor to identify which inlining decision caused the regression
3. Verification - Run RunWithBuildAtHash on commits 8e883c7e4182 (parent) and f21f1945632f with System.Numerics.Tests.Perf_Vector2.NormalizeBenchmark to definitively confirm
4. Cross-platform check - Verify whether x64 platforms also show regression or if this is arm64-specific due to different JIT inlining heuristics

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)	Duration
NormalizeBenchmark	✅ Culprit found	f21f1945632f	✅ RELATED	4.1→5.7 ns (+40%)	4.3→5.7 ns (+32%)	1142s

Total bisection time: 1142s
Success rate: 1/1 tests

System.Numerics.Tests.Perf_Vector2.NormalizeBenchmark

• Culprit: f21f1945632f
• Commit Message: Refactor the various System.Numerics Vector/Matrix/Quaternion/Plane types to require less inlining (Refactor the various System.Numerics Vector/Matrix/Quaternion/Plane types to require less inlining #130274)
• Relevance: RELATED
• Explanation: The commit directly modifies System/Numerics/Vector2.cs, refactoring inlining behavior for Vector2/3/4/Matrix/Quaternion/Plane types. The test Perf_Vector2.NormalizeBenchmark exercises exactly this code path — the "require less inlining" refactor changed how Vector2.Normalize is JIT-compiled, causing a ~33% regression on ARM64.

Helix jobs

Job ID	Submitted by bisection	Last status
958e6e98-4d91-4234-9ce6-d863f4109e9e	✅	submitted
0a259178-7a2c-4620-a137-48254214f219	✅	submitted

Per-commit measurements

Commit	Position	Measurements
c76054f5cb1a	baseline	4.338 ns
c7486c2575a5	3/4	5.750 ns
ce8db36a2e5b	index 5	4.330 ns
8e883c7e4182	index 6	4.339 ns
f21f1945632f	index 7, culprit	5.746 ns
d93f5cd78d48	index 8	5.749 ns
24ed2b075cdb	quarter	4.332 ns
3ade0b55c797	head	5.746 ns
f9a6e2ef41c1	mid	5.750 ns

Bisection details

…[earlier output omitted]

87db11951","abbrevSha":"f...

This is very clearly the culprit. The commit directly modifies System.Numerics.Vector2.cs and is titled "Refactor the various System.Numerics Vector/Matrix/Quaternion/Plane types to require less inlining". The test is System.Numerics.Tests.Perf_Vector2.NormalizeBenchmark — a direct match.

Now let me compile the final results. The measurements clearly show:

• Last good commit: 8e883c7e4182 (4.339 ns)
• Culprit commit: f21f1945632f (5.746 ns)
• Change: +32.4%

BISECTION RESULT:
- Culprit Commit: f21f1945632f432952967797828737587db11951
- Commit Message: Refactor the various System.Numerics Vector/Matrix/Quaternion/Plane types to require less inlining (#130274)
- Commits Tested: 9
- Confidence: 4/5
- Commit Relevance: RELATED
- Relevance Explanation: The commit directly modifies `System/Numerics/Vector2.cs`, refactoring inlining behavior for Vector2/3/4/Matrix/Quaternion/Plane types. The test `Perf_Vector2.NormalizeBenchmark` exercises exactly this code path — the "require less inlining" refactor changed how Vector2.Normalize is JIT-compiled, causing a ~33% regression on ARM64.
- Baseline Value: 4.339 ns   (measured at commit 8e883c7e4182, last good before culprit)
- Culprit Value: 5.746 ns    (measured at commit f21f1945632f, the culprit)
- Summary: Double bisection across 18 commits identified the regression at commit f21f1945632f which refactors System.Numerics vector types to rely less on inlining. The step change from ~4.34 ns to ~5.75 ns is sharp and reproducible, well above the 2σ noise floor of 0.79 ns.

HELIX JOBS:
- 958e6e98-4d91-4234-9ce6-d863f4109e9e
- 0a259178-7a2c-4620-a137-48254214f219

MEASUREMENTS:
- c76054f5cb1a (baseline): 4.338 ns
- 24ed2b075cdb (quarter): 4.332 ns
- ce8db36a2e5b (index 5): 4.330 ns
- 8e883c7e4182 (index 6): 4.339 ns
- f21f1945632f (index 7, culprit): 5.746 ns
- d93f5cd78d48 (index 8): 5.749 ns
- f9a6e2ef41c1 (mid): 5.750 ns
- c7486c2575a5 (3/4): 5.750 ns
- 3ade0b55c797 (head): 5.746 ns

🤖 Proposed Next Actions

Rule: R4_known_regression_new — Known-cause regression — transfer & notify.

Verified regression attributed to commit f21f194 (finding 5/5, regression 5/5, bisection RELATED). No prior linked autofile issue found.

#	Action	Status	Details
1	Back-reference cause PR	⏭️ Skipped (dry-run)	on culprit PR for f21f1945632f
2	Transfer issue	⏭️ Skipped (dry-run)	→ dotnet/runtime
3	Ping owners	⏭️ Skipped (dry-run)	no owners resolved
4	Remove label	⏭️ Skipped (dry-run)	untriaged

🦆 Pre-bisection Re-analysis (advisory)

✅ Agrees with the pre-bisection plan — advisory only, not applied

Rationale: The single candidate has high change confidence (5/5), high finding confidence (4/5), noise=False, a likely commit identified, and relevant stack context — this is a strong signal warranting bisection with no wasted slots.

This critique is advisory in the current configuration and did not change which tests were bisected.

🦆 Re-analysis (advisory)

✅ Agrees with the analysis — advisory only, not applied

Rationale: The bisection converged successfully on the stated culprit with no alignment warning, the history is not bimodal, and the stack relevance is marked as related. Both 5/5 confidence scores are justified by the strong bisection evidence.

This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

---

Generated by PerfTriageAgent on 2026-07-09 11:11 UTC