[PERF][Mono AOT] Linux x64 benchmarks fail with undefined symbol: __udivti3

[LoopedBard3]

Description

The Linux x64 Mono LLVM AOT configuration in dotnet-runtime-perf successfully compiles the benchmarks and BCL, but benchmark processes fail at execution time because the generated System.Private.CoreLib.dll.so and System.Runtime.Numerics.dll.so cannot resolve __udivti3.

In run 20260922.2, build 3083959, this affects partitions 1-14: 39 benchmark processes exit 127, with 28 errors referencing CoreLib and 11 referencing Numerics. Other benchmark processes in those partitions complete successfully. The same 39 benchmark/error pairs recur across 30 examined runs from September 15-22.

Affected operations include Int128 parsing/formatting, BigInteger arithmetic, decimal conversion, and floating-point formatting. This is a native symbol-resolution failure, not the separate result-upload authentication failures in other performance work items.

Reproduction Steps

The reproduction currently available is the existing CI workload; a standalone local reproduction has not been validated.

1. Use the runtime and performance revisions listed under Configuration, with the Linux x64 Mono AOT compiler and custom runtime pack produced by the performance pipeline.

2. Run the performance microbenchmarks using the MonoAOTLLVM toolchain. The recorded toolchain options are:

   --runtimes monoaotllvm
   --aotcompilerpath <payload>/monoaot/mono-aot-cross
   --customruntimepack <payload>/monoaot/pack
   --aotcompilermode llvm
   

   These are options from the failing CI invocation, not a self-contained command. The generated project invokes MonoAOTCompiler with Mode="Normal", OutputType="Library", and UseLLVM=true, and AOT-compiles the published BCL assemblies as well as the benchmark assembly.

3. Inspect x64.micro_mono.net11.0.Partition1 in Helix job 87a4572e-230e-45b2-bda8-777688f84b24. One failing case is System.Tests.Perf_Int128.TryParseSpan with Int128.MinValue as a string. Partition 14 also reproduces the Numerics variant in System.Numerics.Tests.Perf_BigInteger.ToStringD.

The Monitor Helix Jobs log contains the individual console links. These CI links require access to the internal project.

Expected behavior

The generated AOT libraries' compiler-runtime dependencies are resolvable, and these benchmark processes execute and produce results without native loader errors.

Actual behavior

Managed compilation and AOT compilation succeed, but affected benchmark processes terminate with a symbol lookup error. Representative output, with machine-specific paths abbreviated:

<benchmark executable>: symbol lookup error: <publish>/System.Private.CoreLib.dll.so: undefined symbol: __udivti3
ProcessData operation is interrupted by EndOfStream.
ExitCode != 0 and no results reported
No Workload Results were obtained from the run.
// Benchmark Process 52433 has exited with code 127.

The other error variant is:

<benchmark executable>: symbol lookup error: <publish>/System.Runtime.Numerics.dll.so: undefined symbol: __udivti3

The work-item wrapper ultimately exits 1. The errors are visible at lines 31012, 31026, and 31040 of the partition-1 console, and line 20384 of the partition-14 console.

Regression?

Unknown. The earliest directly confirmed occurrence in this investigation is build 3073540, run 20260911.10, queued September 11, 2026 at 22:11 UTC. It has the same 39 failing benchmark/error pairs.

Earlier examined runs were blocked before benchmark execution by NETSDK1045. dotnet/performance#5308 fixed that SDK/target-framework mismatch and exposed this execution failure; it is not established as the cause of the missing symbol.

There is no verified passing execution baseline or first-bad runtime commit. Some historical green builds did not discover their Helix jobs, and the run sequence includes older runtime revisions. The monitor configuration issue has since been fixed.

Known Workarounds

None validated for this configuration.

Configuration

Component	Value
OS / architecture	Ubuntu 22.04, Linux x64
Runtime revision	5edb3a2db2b9ce366aabb208b03ca39ac447ddfb
Reported benchmark runtime	.NET 12.0.0 (12.0.0-ci) using MonoVM
Performance revision	99531dcedc4125a76acc682ca56b1539eace92f6
SDK / benchmark TFM	11.0.100-rc.2.26471.109 / net11.0
BenchmarkDotNet	0.16.0-nightly.20260801.595, source 80844f2e81afe299dc0166d0493c4fbce1158869
Helix queue	ubuntu.2204.amd64.viper.perf

The net11.0 target framework does not identify the runtime being measured: these runs use the custom runtime pack above. Other OS/architecture configurations have not been established as affected.

Other information

Suspected mechanism, not yet a confirmed fix: Mono's LLVM lowering of x64 DivRem emits 128-bit unsigned division/remainder operations, which LLVM can lower to compiler-builtins calls such as __udivti3. The normal Linux AOT linker path uses ld -shared. The observed failure is consistent with a missing builtins dependency, but the generated ELF dependencies and effective native linker invocation still need inspection before selecting a fix.

Some benchmark labels can be misleading: the Int128 benchmark initializer formats Int128.MinValue, so even a nominally unrelated Int128 benchmark can reach the failing formatting path during construction.

Note

This issue was prepared with GitHub Copilot assistance.

[LoopedBard3]

FYI @lewing Not sure if this is something we want to investigate.

[lateralusX]

The most direct short-term mitigation appears to be adjusting BenchmarkDotNet's generated AOT arguments to select Clang for Linux LLVM builds, provided Clang is available on the benchmark agents.

Cause

LLVM can lower operations such as 128-bit division to compiler-builtins calls, including __udivti3. These helpers belong to libgcc/compiler-rt, not libc.

The standalone Linux Mono AOT shared-library path currently defaults to ld -shared, without supplying the required compiler runtime. Shared-library linking permits unresolved references, so compilation can succeed and execution subsequently fail when a helper is called.

BenchmarkDotNet's mcpu=native setting can expose this by enabling intrinsic paths that generate these operations. This affects Normal AOT with LLVM, not just Full AOT.

The standard mobile build flows avoid this problematic default: iOS produces assembly/objects for final linkage through the Apple toolchain; Android uses the NDK toolchain for final linkage or explicitly selects Clang for per-assembly shared libraries. Their compiler-runtime dependencies are handled by the platform toolchain as part of the native build.

Short-term mitigation

BenchmarkDotNet's checked-in src/BenchmarkDotNet/Templates/MonoAOTLLVMCsProj.txt currently sets AotArguments to mcpu=native. It could conditionally add the existing Mono option ld-name=clang:

<AotInputAssemblies Include="$(PublishDirFullPath)\*.dll">
  <AotArguments>mcpu=native</AotArguments>
  <AotArguments Condition="'$USELLVM$' == 'true' and $([MSBuild]::IsOSPlatform('Linux'))">
    mcpu=native;ld-name=clang
  </AotArguments>
</AotInputAssemblies>

$USELLVM$ is already substituted by BenchmarkDotNet's generator. This preserves the existing behavior for non-LLVM and non-Linux configurations. Clang then supplies the compiler-runtime libraries and search paths when linking the generated objects.

Clang availability should be documented as a prerequisite for this path; having the Mono AOT compiler and packaged LLVM tools does not itself provide a Clang executable.

For callers that must retain ld, the existing ld-flags option provides another mitigation:

<AotArguments>mcpu=native;ld-flags=/path/to/target/libgcc.a</AotArguments>

Mono appends these flags after the generated objects, allowing ordinary archive extraction to include only the required members. No --whole-archive is needed. The caller must locate the correct target archive; for a native GCC toolchain, gcc -print-libgcc-file-name reports its location.

Longer-term direction

Mono could default to the appropriate target compiler driver for LLVM-enabled Linux shared-library builds, preserving explicit overrides and cross-compilation settings. This should not be restricted to Full AOT or mcpu=native.

Alternatively, a bundled static solution would require packaging and linking target-specific compiler-rt builtins. The Linux x64 LLVM SDK/Tools packages used here do not currently contain a builtins archive, so there is no existing packaged library we can simply add to the link command.

@LoopedBard3, would changing the linker driver to clang work to mitigate this issue on your end?

[dotnet-policy-service]

This issue has been marked needs-author-action and may be missing some important information.