[Perf] Linux/x64: 1 Regression on 9/15/2026 9:27:19 PM +00:00

[performanceautofiler]

Run Information

Name	Value
Architecture	x64
OS	ubuntu 22.04
Queue	ViperUbuntu
Baseline	6af607b8858d946ddbf4101a18b98de61c2eb5e1
Compare	76e6281d52fb24edd205b3959090189b9c887f56
Diff	Diff
Configs	CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.CreateAddAndClear<String>

Benchmark	Baseline	Test	Test/Base	Test Quality	Edge Detector	Baseline IR	Compare IR	IR Ratio
ConcurrentStack - Duration of single invocation 📝 - Benchmark Source ADX - Test Multi Config Graph	4.81 μs	5.44 μs	1.13	0.22	False

Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.CreateAddAndClear<String>*'

Details

System.Collections.CreateAddAndClear<String>.ConcurrentStack(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

[LoopedBard3]

🔍 Automated Triage Analysis

Summary: Verified regression in ConcurrentStack(Size: 512) — caused by commit 64878f5.

Finding Confidence: 5/5 (analysis accuracy)
Regression Confidence: 5/5 (likelihood of true regression)

Likely Cause: 64878f521de0dcb92dcfb0ab623481619aa6eeab

🤖 Proposed Next Actions: Known-cause regression — transfer & notify — 0 done, 3 dry-run (rule R4_known_regression_new — see Full Analysis for the table).

📊 Full Analysis (click to expand)

Summary: No statistically confirmed regressions were identified; the sole reported test is a borderline +13.08% movement that exceeds its 10.19% adaptive threshold but remains only 0.93σ above the historical mean after a downward baseline drift. Commit 64878f5 (CoreCLR PAL waits) is the leading candidate because the change landed at the regression timestamp and sibling issue #79728 reports the same ConcurrentStack pattern at that boundary, but the mechanism is indirect; finding confidence is 4/5 and regression confidence is 3/5.

Issue Overview

• Issue: #79729 - [Perf] Linux/x64: 1 Regression on 9/15/2026 9:27:19 PM +00:00
• Date: 2026-09-15 | Queue: Ubuntu.2204.Amd64.Viper.Perf | OS/Arch: Ubuntu 22.04/x64
• Commits: 6af607b → 76e6281 (14 commits)
• Labels: untriaged, branch-refs/heads/main, perf-regression, arch-x64, os-linux, runtime-coreclr, kind-micro, compilationmode-tiered, runkind-micro
• Runtime stack: Desktop + CoreCLR
• Description: One Linux/x64 regression was reported for System.Collections.CreateAddAndClear<String>.ConcurrentStack(Size: 512).
• Historical Data: Available (293 prior runs; 315 total runs examined, including 22 after the regression date)
• Benchmark Source: CreateAddAndClear.cs lines 192-202

Test Analysis

Test Name	Baseline	Compare	Δ%	μ	σ	n	z	Thresh%	Noise?	Assessment
System.Collections.CreateAddAndClear<String>.ConcurrentStack(Size: 512)	4.809 μs	5.438 μs	+13.08%	5.191 μs	0.265 μs	293	0.932	10.19%	N	Borderline: threshold exceeded, but compare is within historical variance and baseline followed a downward drift

Legend: μ=mean, σ=stddev, n=prior runs, z=z-score, Thresh%=adaptive threshold

Finding Confidence: 4/5 - The historical series is comprehensive and the complete 14-commit window was reviewed. Confidence is not 5/5 because no isolated rerun or bisection result is available.

Candidate Stack Relevance

Commit	Touched Paths (summary)	Relevance	Notes
64878f5	src/coreclr/runtime/, src/coreclr/pal/, src/coreclr/gc/, debugger/runtime wait callers	Relevant	Replaced the Unix CoreCLR PAL wait subsystem; exact regression timestamp and same benchmark family moved in sibling #79728
0554cde	src/coreclr/jit/compiler.hpp	Relevant	Surgical safepoint/interruptibility change, but only affects loops containing special intrinsics and has no demonstrated connection to this push/clear loop
e173d24	src/coreclr/jit/optimizer.cpp	Relevant	Changes loop memory-havoc treatment around volatile reads; SuperPMI reported only two changed compilations and no direct benchmark match
76e6281	src/coreclr/vm/interpexec.cpp, src/tests/async/	LessLikely	Wasm async continuation fix; the Desktop/CoreCLR tiered benchmark does not exercise the Wasm interpreter path

Confirmed Regressions

System.Collections.CreateAddAndClear<String>.ConcurrentStack(Size: 512) (+13.08%, borderline)

• Delta: 4.809 μs → 5.438 μs (+13.08%)
• Historical Context: μ=5.191 μs (σ=0.265 μs, n=293); z-score=0.932; adaptive threshold=10.19%; CV=5.10%
• Assessment: Gradual drift / borderline baseline-relative regression. The delta exceeds the adaptive threshold, but the compare value is not a historical outlier and the low baseline followed a downward drift.
• Finding Confidence: 4/5 - Ample historical data and a complete commit summary support the statistical interpretation and candidate review; no verification run was performed.
• Regression Confidence: 3/5 - The +13.08% delta exceeds the adaptive threshold and is corroborated by sibling issue #79728, which reports ConcurrentStack and ImmutableStack regressions at the same suspected boundary. Confidence is limited by z=0.932, CV=5.10%, pre-existing drift, and the lack of a direct hot-path code change.
• Likely Cause: 64878f5 - "Replace CoreCLR PAL wait subsystem"
  • Affected Areas: CoreCLR PAL, Unix waits, runtime/GC synchronization
  • Stack Relevance: Relevant
  • Rationale: This large Linux-relevant CoreCLR change landed at the recorded regression timestamp and sibling issue #79728 independently places the same CreateAddAndClear.ConcurrentStack movement at this commit boundary. The plausible mechanism is indirect runtime interference: changing Unix event, sleep, thread-exit, and GC/runtime wait behavior can alter background-thread wakeups or scheduling during a short microbenchmark, but it does not modify ConcurrentStack<T>.Push, Clear, or the benchmark's generated code. This makes 64878f5 the strongest candidate, not a verified root cause.
  • Candidates Considered:
    • 0554cde - A three-line JIT safepoint change is focused, but the benchmark loop has no identified removable special intrinsic.
    • e173d24 - Volatile-read loop-hoisting changes could theoretically affect concurrent collection code, but the changed optimization is narrowly scoped and no direct compilation impact was established.
    • 76e6281 - The compare commit is a Wasm-specific async stack-walk fix and is not exercised by this Desktop/CoreCLR run.
• Verification Status: Not verified

Open Questions

• Historical Data: Sufficient (293 prior runs), but CV=5.10% and a downward baseline drift conflict with the baseline-relative regression signal.
• Confidence Limitations: No commit directly changes ConcurrentStack or its benchmark hot path; attribution relies on timing, Linux/CoreCLR stack relevance, and sibling correlation.
• Verification Needed: Compare 0554cde (the parent) with 64878f5 in an isolated rerun. Include the String case from this issue and the Int32 ConcurrentStack/ImmutableStack cases from sibling #79728 to test the shared-boundary hypothesis.
• Bisection Status: Not performed

Recommended Actions

1. Boundary verification - Run System.Collections.CreateAddAndClear<String>.ConcurrentStack(Size: 512) at 0554cde and 64878f5 with RunWithBuildAtHash.
2. Corroborate the shared pattern - In the same run, include the Int32 ConcurrentStack and ImmutableStack tests from #79728; synchronized movement would strengthen the PAL/runtime-interference hypothesis.
3. Profile only if reproduced - Collect hardware counters and runtime/GC scheduling data around 64878f5, focusing on wakeups, suspension activity, and context switching rather than ConcurrentStack<T> implementation changes.
4. Monitor subsequent runs - Check whether values remain near the post-change level or return toward the low baseline, as recommended by the historical analysis.

Bisection Results

Test	Status	Culprit Commit	Relevance	Reported Diff (issue)	Bisected Diff (at culprit)
ConcurrentStack(Size: 512)	✅ Culprit found	64878f521de0	✅ RELATED	4.8→5.4 μs (+13%)	4.7→5.2 μs (+10%)

Total bisection time: 23m 22s
Tests measured: 1/1
Culprit found: 1/1

System.Collections.CreateAddAndClear<String>.ConcurrentStack(Size: 512)

• Culprit: 64878f521de0
• Relevance: RELATED
• Explanation: Finalized culprit 64878f5 matches the pre-bisection analysis likely cause 64878f5, which was classified Relevant to System.Collections.CreateAddAndClear<String>.ConcurrentStack(Size: 512). The analysis relevance is inherited because it was computed with the full commit/path context.

Bisection details

• Culprit: 64878f521de0
• Relevance: RELATED
• Baseline: 4.809 μs
• Culprit value: 5.191 μs

Per-commit measurements (oldest → newest) (μs)

#	Commit	Result	Measurement
0	6af607b8858d	baseline/good	4.813 μs
5	e173d2476020	good	4.758 μs
9	6905a9f4294a	good	4.748 μs
11	0554cde34f84	good	4.727 μs
12	64878f521de0	culprit/bad	5.191 μs
14	76e6281d52fb	head/bad	5.083 μs

Helix jobs

Job ID
9c9fa7e2-5000-43a8-aa2c-b22ffdca09a3
1c809e5b-e736-442c-85ce-e463a2aca931

🤖 Proposed Next Actions

Rule: R4_known_regression_new — Known-cause regression — transfer & notify.

Verified regression attributed to commit 64878f5 (finding 5/5, regression 5/5, bisection RELATED). No prior linked autofile issue found.

#	Action	Status	Details
1	Back-reference cause PR	⏭️ Skipped (dry-run)	on culprit PR for 64878f521de0
2	Transfer issue	⏭️ Skipped (dry-run)	→ dotnet/runtime
3	Ping owners	⏭️ Skipped (dry-run)	ping jkoritzinsky

🦆 Pre-bisection Re-analysis (advisory)

✅ Agrees with the pre-bisection plan — advisory only, not applied

Rationale: The sole finding is classified as actionable with moderate change confidence and relevant stack evidence, with no reported bimodality or clear noise indicator. Queuing only this test is appropriately selective.

This critique is advisory in the current configuration and did not change which tests were bisected.

🦆 Re-analysis (advisory)

⚠️ Disputes the analysis — advisory only, not applied

Flags: overconfident, insufficient-signal-evidence

Rationale: The successful, RELATED bisection supports the culprit and actionable classification, but the aggregate 5/5 scores exceed the sole finding's 4/5 finding confidence and 3/5 change confidence. No z-score, variance, or sustained after-values are provided to justify raising either score.

Suggested confidence: Finding Confidence → 4/5; Change Confidence → 3/5 (advisory — not applied)

This critique is advisory in the current configuration and did not change the confidences, culprit, or proposed actions above.

---

Generated by PerfTriageAgent on 2026-09-22 09:51 UTC

[LoopedBard3]

Bisection found: #132927 as the culprit, however it only seems to be impacting this test on Linux X64. FYI @jkoritzinsky.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-collections
See info in area-owners.md if you want to be subscribed.

[LoopedBard3]

Related Regressions:

• [Perf] Linux/x64: 2 Regressions on 9/15/2026 3:09:02 PM +00:00 perf-autofiling-issues#79728 (Possibly related though not a strong signal)