[Experiment] Use SIMD-friendly non-randomized string hashing

[EgorBo]

This PR changes the hash implementation for the following internal APIs:

GetNonRandomizedHashCode()
GetNonRandomizedHashCodeOrdinalIgnoreCase()
GetNonRandomizedHashCode(ReadOnlySpan<char> span)
GetNonRandomizedHashCodeOrdinalIgnoreCase(ReadOnlySpan<char> span)

via one SIMD-friendly implementation, fully memory safe.

The previous one duplicated (not even sure GetNonRandomizedHashCode() and GetNonRandomizedHashCode(ReadOnlySpan<char> span) were the same). Relied on unsafe code + did intentional out of bounds read with the assumption about string's zero termnator char. And most importantly, were not SIMD friendly.

The hash code impl should not change its output depending on whether SIMD is enabled or disabled.

Throughput

Dictionary<string, int> lookup by an equal-but-not-reference-equal key (benchmark, full results). Numbers are speedup vs main, higher is better. The non-ASCII columns use the same key with a single Cyrillic letter in the middle, which is what forces the ignore-case slow path.

StringComparer.Ordinal

Key length	x64 Zen 5 ASCII	x64 Zen 5 non-ASCII	Arm64 Apple M4 ASCII	Arm64 Apple M4 non-ASCII	Arm64 Neoverse N2 ASCII	Arm64 Neoverse N2 non-ASCII
1	1.00x	1.04x	0.93x	0.97x	1.08x	1.08x
4	0.91x	0.91x	1.01x	1.00x	1.00x	0.97x
10	0.93x	0.96x	0.99x	1.00x	0.93x	0.89x
16	1.10x	1.08x	1.10x	1.09x	1.02x	1.01x
20	0.98x	0.96x	0.94x	0.94x	0.95x	0.93x
50	1.37x	1.36x	1.23x	1.24x	1.10x	1.09x
100	1.76x	1.76x	1.55x	1.55x	1.26x	1.25x
1000	1.79x	1.78x	1.74x	1.74x	1.69x	1.70x
16000	1.81x	1.82x	1.85x	1.85x	1.77x	1.78x
geomean	1.24x	1.25x	1.22x	1.22x	1.17x	1.15x

StringComparer.OrdinalIgnoreCase

Key length	x64 Zen 5 ASCII	x64 Zen 5 non-ASCII	Arm64 Apple M4 ASCII	Arm64 Apple M4 non-ASCII	Arm64 Neoverse N2 ASCII	Arm64 Neoverse N2 non-ASCII
1	1.05x	1.49x	1.14x	1.04x	1.25x	1.14x
4	0.98x	0.96x	1.12x	1.04x	1.11x	1.03x
10	1.10x	0.92x	1.21x	1.00x	1.19x	1.00x
16	1.22x	0.97x	1.40x	1.03x	1.33x	1.01x
20	1.13x	0.92x	1.22x	0.94x	1.20x	0.97x
50	1.71x	1.02x	1.52x	1.04x	1.64x	1.06x
100	1.87x	1.27x	1.91x	1.10x	1.91x	1.14x
1000	1.86x	1.19x	1.77x	1.21x	2.51x	1.22x
16000	1.90x	1.15x	1.75x	1.19x	2.64x	1.28x
geomean	1.37x	1.08x	1.42x	1.06x	1.56x	1.09x

Note

Three microarchitectures are shown because the block loop is latency-bound: its recurrence costs chain_depth × vector-ALU-latency, and that latency is 1 cycle on Zen 4 but 2 cycles on Zen 5, Apple silicon and Neoverse N2. An earlier revision that ignored this was up to 1.38x slower than main on long strings despite being faster on Zen 4. The loop now runs two independent accumulators over 32-byte blocks with a 3-deep recurrence, so it wins on both.

The worst case anywhere is 0.89x, under two nanoseconds, in the 4-20 character range where main's plain 8-byte loop is hard to beat. Only compare within a column: these runs come from shared machines and the absolute timings move between runs.

Collision analysis

Verified against the real implementations (bound by reflection, so the analyzer runs unmodified on both builds) rather than a model, over 24 key sets x 200k keys: random at many lengths, counters, prefix+counter, suffix+counter, path-like keys, dotted type names, GUIDs, hex, single-character differences, sparse bit differences, repeated characters, Cyrillic and CJK, and case variants. A reference-quality hash (FNV-1a + murmur3 finalizer) is included as a control, so a weak hash can be told apart from a bad measurement.

Metric	main	this PR	control	ideal
Probe length, prime buckets (worst)	1.55	1.52	1.51	~1.50
Probe length, power-of-two buckets (geomean)	9.14	1.83	1.37	1.38
Probe length, power-of-two buckets (worst)	913.6	8.9	1.4	1.38
Full 32-bit collisions, all sets	6764	236	120	~120
Avalanche, mean bias	0.276	0.166	0.009	0.009

Probe length is the number of chain entries visited for a successful lookup. Against the prime modulus that Dictionary and HashSet actually use, main and this PR are equivalent, so real dictionary behaviour does not regress. The difference is in the low bits, which matters for anything that masks instead of taking a remainder, and there this PR is a large improvement.

main collides whole families of realistic keys: single-character differences at a fixed position degrade its power-of-two probe length to 913, and prefix + counter keys to 125.

Neither hash is avalanche-quality, by design; both are speed-first, and the prime modulus is what keeps that safe.

One collision family found while measuring was specific to this PR and has been fixed: combining the length into the hash linearly cancels against a character whose delta matches the length delta, so first ^ (uint)length collided every pair like "600"/"8000" ('6' ^ 6 == '8' ^ 8) with an identical trailing word. That was 507 collisions over 200k decimal keys where main has none. Multiplying the length by a prime before mixing leaves 3, against ~4.7 expected for a random 32-bit hash, and costs one instruction on the 4-character path and nothing elsewhere.

Behaviour on very long inputs

MemoryMarshal.AsBytes throws OverflowException once a char span exceeds int.MaxValue / 2 elements, so working in bytes needs care. The entry points route anything longer through a chunked path, and the ignore-case slow path slices its scratch buffer to the length actually needed. Checked either side of that boundary with the non-ASCII character first, in the middle and last, for both comparers: no exception in any of the twelve combinations.

main throws in six of them, from Ascii.ChangeCase handing MemoryMarshal.AsBytes an unsliced ArrayPool buffer whose length can exceed the limit. That is a pre-existing issue independent of this PR.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-runtime
See info in area-owners.md if you want to be subscribed.

[Copilot]

🟡 Changes recommended

Multiple moderate issues require restored coverage, SIMD/scalar validation, and a less costly non-ASCII fallback.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Replaces duplicated non-randomized string hashing with a shared, memory-safe SIMD implementation.

Changes:

• Adds unified ordinal and ordinal-ignore-case hashing.
• Removes the previous unsafe implementations.
• Removes several comparer, hash-equivalence, and collision-transition tests.

File summaries

File	Review
src/libraries/System.Runtime/tests/System.Runtime.Tests/System/StringTests.cs	Removes string/span hash-equivalence coverage.
src/libraries/System.Runtime/tests/System.Runtime.Tests/System/StringComparerTests.cs	Restore reflected internal comparer cases and helper to preserve unwrapping-branch coverage. Moderate; 2 votes.
src/libraries/System.Private.CoreLib/src/System/String.NonRandomizedHashCode.cs	Add SIMD/scalar parity and bounded-memory tests; avoid the redundant non-ASCII source buffer; provide representative benchmarks. Two moderate and one nit; 2 votes each.
src/libraries/System.Private.CoreLib/src/System/String.Comparison.cs	Removes the previous hash implementations.
src/libraries/System.Private.CoreLib/src/System.Private.CoreLib.Shared.projitems	Includes the new hash implementation source.
src/libraries/System.Collections/tests/Generic/Dictionary/HashCollisionScenarios/OutOfBoundsRegression.cs	Replace, rather than remove, collision-threshold and comparer-upgrade coverage. Moderate; 2 votes.
src/libraries/System.Collections.Concurrent/tests/ConcurrentDictionary/ConcurrentDictionary.Generic.Tests.cs	Retain equivalent coverage for randomized-comparer upgrades during collisions. Moderate; 2 votes.

Review details

• Files reviewed: 7/7 changed files
• Comments generated: 6
• Review effort level: Balanced

[EgorBo]

@EgorBot -linux_amd -osx_arm64 -linux_arm64

using System;
using System.Collections.Generic;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[MemoryDiagnoser]
public class Benchmarks {
    private Dictionary<string, int> _dictionary = null!;
    private string _key = null!;

    [Params(1, 4, 10, 16, 20, 50, 100, 1000, 16000)]
    public int Length { get; set; }

    [GlobalSetup(Target = nameof(Ordinal))]
    public void SetupOrdinal() => Setup(StringComparer.Ordinal);

    [GlobalSetup(Target = nameof(OrdinalIgnoreCase))]
    public void SetupOrdinalIgnoreCase() =>
        Setup(StringComparer.OrdinalIgnoreCase);

    private void Setup(StringComparer comparer) {
        _key = new string('a', Length);
        _dictionary = new Dictionary<string, int>(1024, comparer) {
            [new string(_key.AsSpan())] = 42
        };
    }

    [Benchmark]
    public int Ordinal() => _dictionary[_key];

    [Benchmark]
    public int OrdinalIgnoreCase() => _dictionary[_key];
}

[EgorBo]

@EgorBot -linux_amd -osx_arm64 -linux_arm64

using System;
using System.Collections.Generic;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(Benchmarks).Assembly).Run(args);

[MemoryDiagnoser]
public class Benchmarks {
    private Dictionary<string, int> _dictionary = null!;
    private string _key = null!;

    [Params(1, 4, 10, 16, 20, 50, 100, 1000, 16000)]
    public int Length { get; set; }

    [GlobalSetup(Target = nameof(Ordinal))]
    public void SetupOrdinal() => Setup(StringComparer.Ordinal);

    [GlobalSetup(Target = nameof(OrdinalIgnoreCase))]
    public void SetupOrdinalIgnoreCase() =>
        Setup(StringComparer.OrdinalIgnoreCase);

    private void Setup(StringComparer comparer) {
        _key = new string('a', Length);
        _dictionary = new Dictionary<string, int>(1024, comparer) {
            [new string(_key.AsSpan())] = 42
        };
    }

    [Benchmark]
    public int Ordinal() => _dictionary[_key];

    [Benchmark]
    public int OrdinalIgnoreCase() => _dictionary[_key];
}

[EgorBo]

@EgorBot -linux_amd -osx_arm64 -linux_arm64

using System;
using System.Collections.Generic;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(ASCIIBenchmarks).Assembly).Run(args);

[MemoryDiagnoser]
public class ASCIIBenchmarks {
    private Dictionary<string, int> _dictionary = null!;
    private string _key = null!;

    [Params(1, 4, 10, 16, 20, 50, 100, 1000, 16000)]
    public int Length { get; set; }

    [GlobalSetup(Target = nameof(Ordinal))]
    public void SetupOrdinal() => Setup(StringComparer.Ordinal);

    [GlobalSetup(Target = nameof(OrdinalIgnoreCase))]
    public void SetupOrdinalIgnoreCase() => Setup(StringComparer.OrdinalIgnoreCase);

    private void Setup(StringComparer comparer) {
        _key = new string('a', Length);
        _dictionary = new Dictionary<string, int>(1024, comparer) {
            [new string(_key.AsSpan())] = 42
        };
    }

    [Benchmark]
    public int Ordinal() => _dictionary[_key];

    [Benchmark]
    public int OrdinalIgnoreCase() => _dictionary[_key];
}

[MemoryDiagnoser]
public class NonASCIIBenchmarks {
    private Dictionary<string, int> _dictionary = null!;
    private string _key = null!;

    [Params(1, 4, 10, 16, 20, 50, 100, 1000, 16000)]
    public int Length { get; set; }

    [GlobalSetup(Target = nameof(Ordinal))]
    public void SetupOrdinal() => Setup(StringComparer.Ordinal);

    [GlobalSetup(Target = nameof(OrdinalIgnoreCase))]
    public void SetupOrdinalIgnoreCase() => Setup(StringComparer.OrdinalIgnoreCase);

    private void Setup(StringComparer comparer) {
        // One Cyrillic letter in the middle, the rest ASCII.
        char[] chars = new char[Length];
        Array.Fill(chars, 'a');
        chars[Length / 2] = '\u0416';
        _key = new string(chars);
        _dictionary = new Dictionary<string, int>(1024, comparer) {
            [new string(_key.AsSpan())] = 42
        };
    }

    [Benchmark]
    public int Ordinal() => _dictionary[_key];

    [Benchmark]
    public int OrdinalIgnoreCase() => _dictionary[_key];
}

[EgorBo]

@EgorBot -linux_amd -osx_arm64 -linux_arm64

using System;
using System.Collections.Generic;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkSwitcher.FromAssembly(typeof(ASCIIBenchmarks).Assembly).Run(args);

[MemoryDiagnoser]
public class ASCIIBenchmarks {
    private Dictionary<string, int> _dictionary = null!;
    private string _key = null!;

    [Params(1, 4, 10, 16, 20, 50, 100, 1000, 16000)]
    public int Length { get; set; }

    [GlobalSetup(Target = nameof(Ordinal))]
    public void SetupOrdinal() => Setup(StringComparer.Ordinal);

    [GlobalSetup(Target = nameof(OrdinalIgnoreCase))]
    public void SetupOrdinalIgnoreCase() => Setup(StringComparer.OrdinalIgnoreCase);

    private void Setup(StringComparer comparer) {
        _key = new string('a', Length);
        _dictionary = new Dictionary<string, int>(1024, comparer) {
            [new string(_key.AsSpan())] = 42
        };
    }

    [Benchmark]
    public int Ordinal() => _dictionary[_key];

    [Benchmark]
    public int OrdinalIgnoreCase() => _dictionary[_key];
}

[MemoryDiagnoser]
public class NonASCIIBenchmarks {
    private Dictionary<string, int> _dictionary = null!;
    private string _key = null!;

    [Params(1, 4, 10, 16, 20, 50, 100, 1000, 16000)]
    public int Length { get; set; }

    [GlobalSetup(Target = nameof(Ordinal))]
    public void SetupOrdinal() => Setup(StringComparer.Ordinal);

    [GlobalSetup(Target = nameof(OrdinalIgnoreCase))]
    public void SetupOrdinalIgnoreCase() => Setup(StringComparer.OrdinalIgnoreCase);

    private void Setup(StringComparer comparer) {
        // One Cyrillic letter in the middle, the rest ASCII.
        char[] chars = new char[Length];
        Array.Fill(chars, 'a');
        chars[Length / 2] = '\u0416';
        _key = new string(chars);
        _dictionary = new Dictionary<string, int>(1024, comparer) {
            [new string(_key.AsSpan())] = 42
        };
    }

    [Benchmark]
    public int Ordinal() => _dictionary[_key];

    [Benchmark]
    public int OrdinalIgnoreCase() => _dictionary[_key];
}