Optimize HashSet intersecting

[pentp]

Optimize HashSet<T> intersection/subset/overlap calculations, especially for worst case scenarios.

HashSet<int>	Overlap	Mean (main)	Error	Mean (PR)	Error
IntersectWithEnumerable	0	19,929.2 ns	296.9 ns	6,023.0 ns	33.2 ns
IntersectWithEnumerable	50	13,488.4 ns	125.7 ns	5,431.6 ns	105.3 ns
IntersectWithEnumerable	100	5,198.2 ns	36.2 ns	4,485.3 ns	51.5 ns
IntersectWithHashSetWithSameComparer	0	3,799.0 ns	29.6 ns	3,192.8 ns	33.0 ns
IntersectWithHashSetWithSameComparer	50	2,993.8 ns	31.3 ns	2,759.8 ns	44.4 ns
IntersectWithHashSetWithSameComparer	100	2,317.4 ns	21.9 ns	2,305.7 ns	20.4 ns
IsSupersetOfHashSetWithSameComparer	0	89.5 ns	1.0 ns	36.7 ns	0.4 ns
IsSupersetOfHashSetWithSameComparer	50	144.0 ns	1.8 ns	42.6 ns	0.6 ns
IsSupersetOfHashSetWithSameComparer	100	16,266.0 ns	188.2 ns	1,328.3 ns	14.2 ns
OverlapsHashSetWithSameComparer	0	1,500.0 ns	10.6 ns	1,125.5 ns	11.7 ns
OverlapsHashSetWithSameComparer	50	6.6 ns	0.1 ns	42.0 ns	0.4 ns
OverlapsHashSetWithSameComparer	100	6.6 ns	0.2 ns	21.3 ns	0.3 ns
SymmetricExceptWithEnumerable	0	7,691.9 ns	90.7 ns	7,391.4 ns	119.7 ns
SymmetricExceptWithEnumerable	50	15,198.2 ns	129.9 ns	8,040.1 ns	141.0 ns
SymmetricExceptWithEnumerable	100	20,029.7 ns	331.0 ns	6,052.9 ns	33.7 ns

HashSet<string>	Overlap	Mean (main)	Error	Mean (PR)	Error
IntersectWithEnumerable	0	37,776.2 ns	330.0 ns	9,290.5 ns	80.1 ns
IntersectWithEnumerable	50	26,304.8 ns	319.6 ns	9,467.2 ns	121.5 ns
IntersectWithEnumerable	100	10,489.1 ns	147.4 ns	8,645.8 ns	92.0 ns
IntersectWithHashSetWithSameComparer	0	10,848.6 ns	186.7 ns	3,421.3 ns	45.7 ns
IntersectWithHashSetWithSameComparer	50	9,199.0 ns	159.9 ns	3,330.9 ns	47.2 ns
IntersectWithHashSetWithSameComparer	100	7,353.5 ns	112.7 ns	3,428.5 ns	28.7 ns
IsSupersetOfHashSetWithSameComparer	0	127.8 ns	2.4 ns	27.7 ns	0.4 ns
IsSupersetOfHashSetWithSameComparer	50	222.6 ns	1.8 ns	34.3 ns	0.4 ns
IsSupersetOfHashSetWithSameComparer	100	28,581.9 ns	402.8 ns	2,150.9 ns	26.5 ns
OverlapsHashSetWithSameComparer	0	5,600.5 ns	92.6 ns	1,295.9 ns	14.7 ns
OverlapsHashSetWithSameComparer	50	24.7 ns	0.5 ns	41.9 ns	0.6 ns
OverlapsHashSetWithSameComparer	100	27.2 ns	0.3 ns	17.6 ns	0.2 ns
SymmetricExceptWithEnumerable	0	13,148.2 ns	246.2 ns	11,521.7 ns	224.2 ns
SymmetricExceptWithEnumerable	50	27,093.3 ns	168.6 ns	13,285.7 ns	109.3 ns
SymmetricExceptWithEnumerable	100	40,047.7 ns	508.4 ns	11,195.2 ns	151.7 ns

Benchmark code

[GenericTypeArguments(typeof(int))] // value type
[GenericTypeArguments(typeof(string))] // reference type
public class Bench<T>
{
    public const int Size = 512;

    [Params(0, 50, 100)]
    public int Overlap;

    private HashSet<T> _mainSet;
    private T[][] _otherKeys;
    private HashSet<T>[] _otherSets;

    [GlobalSetup]
    public void Setup()
    {
        var all = ValuesGenerator.ArrayOfUniqueValues<T>(Size * 2);
        var keys = all.AsSpan(0, Size).ToArray();
        _mainSet = new(keys);

        var keys0 = all.AsSpan(Size, Size).ToArray();
        var rnd = new Random(42);
        var keys100 = keys.AsSpan().ToArray();
        rnd.Shuffle(keys100);

        rnd.Shuffle(all);
        var keys50 = all.AsSpan(0, Size).ToArray();

        _otherKeys = [keys0, keys50, keys100];
        _otherSets = Array.ConvertAll(_otherKeys, x => new HashSet<T>(x));
    }

    [Benchmark]
    public HashSet<T> IntersectWithHashSetWithSameComparer()
    {
        var hashSet = new HashSet<T>(_mainSet);
        hashSet.IntersectWith(_otherSets[Overlap / 50]);
        return hashSet;
    }

    [Benchmark]
    public HashSet<T> IntersectWithEnumerable()
    {
        var hashSet = new HashSet<T>(_mainSet);
        hashSet.IntersectWith(_otherKeys[Overlap / 50]);
        return hashSet;
    }

    [Benchmark]
    public HashSet<T> SymmetricExceptWithEnumerable()
    {
        var hashSet = new HashSet<T>(_mainSet);
        hashSet.SymmetricExceptWith(_otherKeys[Overlap / 50]);
        return hashSet;
    }

    [Benchmark]
    public bool IsSupersetOfHashSetWithSameComparer() => _mainSet.IsSupersetOf(_otherSets[Overlap / 50]);

    [Benchmark]
    public bool OverlapsHashSetWithSameComparer() => _mainSet.Overlaps(_otherSets[Overlap / 50]);
}

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-collections
See info in area-owners.md if you want to be subscribed.

[Copilot]

Pull request overview

This PR optimizes HashSet<T> set operations (intersection, subset, overlap, superset, symmetric except) by reducing unnecessary work in multiple ways: reusing stored hashcodes via a new Contains(ref Entry) overload, replacing Remove(value) with a more efficient RemoveAt(entries, index), using BitHelper.FindFirstUnmarked() to skip already-processed entries, and adding fast paths for Overlaps and IsSupersetOf when the other set is a HashSet<T> with compatible comparers. It also simplifies HashSetEqualityComparer by delegating to SetEquals.

Changes:

• Added Contains(ref Entry), RemoveAt, OverlapsHashSetWithSameComparer, and enhanced IsSubsetOfHashSetWithSameComparer/IntersectWithHashSetWithSameComparer to reuse pre-computed hashcodes when effective comparers match, falling back to standard Contains(value) otherwise.
• Added BitHelper.TryMarkBit, IsUnmarked, FindFirstUnmarked methods and optimized ToIntArrayLength to support more efficient bit-marking operations in set algorithms.
• Simplified HashSetEqualityComparer.Equals to delegate to SetEquals, changed EqualityComparersAreEqual/EffectiveEqualityComparersAreEqual to include fast reference-equality checks, and removed the EffectiveComparer property.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
src/libraries/Common/src/System/Collections/Generic/BitHelper.cs	Added TryMarkBit, IsUnmarked, FindFirstUnmarked methods; optimized ToIntArrayLength to use uint arithmetic
src/libraries/System.Private.CoreLib/src/System/Collections/Generic/HashSet.cs	Core optimizations: new Contains(ref Entry) and RemoveAt overloads, optimized IsSupersetOf/Overlaps fast paths, enhanced internal methods to reuse hashcodes, simplified comparer equality checks
src/libraries/System.Private.CoreLib/src/System/Collections/Generic/HashSetEqualityComparer.cs	Simplified Equals to delegate to x.SetEquals(y) instead of manual O(N²) comparison

[Copilot]

Changing from the old per-element comparison with EqualityComparer<T>.Default to x.SetEquals(y) introduces a behavioral change for HashSetEqualityComparer when the two sets use different comparers.

Previously, elements were compared using EqualityComparer<T>.Default regardless of either set's comparer. Now, x.SetEquals(y) uses x's comparer. This can produce different results and also breaks the IEqualityComparer<T> contract: Equals(x, y) may not equal Equals(y, x) when x and y have different comparers, and GetHashCode (which still uses T.GetHashCode()) may be inconsistent with Equals when a custom comparer considers elements equal that the default comparer doesn't.

That said, the old code also had a bug: the O(N²) loop only checked that every element of y was in x, without checking the reverse, so Equals could return true for sets of different sizes. The new code is more correct in that regard. If this behavioral change is intentional, it would be worth documenting in the PR description.

[pentp]

It's probably fundamentally impossible to make this class work with mismatching comparers. Like Copilot says, the old behavior was pretty broken and SetEquals doesn't fully solve it. We could consider changing it to only consider sets with matching comparers (and change GetHashCode also then - for example currently ignore-case sets will have mismatching hashcodes even if the contents differ only in case).

[pentp]

@stephentoub what do you think would be most appropriate here?

[pentp]

I removed HashSetEqualityComparer changes from this PR - probably best addressed in a separate issue/PR.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

[github-actions]

Workflow state for the Holistic Review Orchestrator.

{
  "version": 5,
  "last_dispatched_commit": "740c77105766feae0f008a67308e6ee4a097e3de",
  "last_dispatched_base_ref": "main",
  "last_dispatched_base_sha": "8b06ce0e84bcbcbd16ae4c9cbcdd3959fbbe498a",
  "last_reviewed_commit": "740c77105766feae0f008a67308e6ee4a097e3de",
  "last_reviewed_base_ref": "main",
  "last_reviewed_base_sha": "8b06ce0e84bcbcbd16ae4c9cbcdd3959fbbe498a",
  "last_recorded_worker_run_id": "29675331201",
  "review_attempt_commit": "",
  "review_attempt_base_ref": "",
  "review_attempt_count": 0,
  "max_review_attempts": 5,
  "review_history_format": "holistic-review-disclosure-v1",
  "review_history": [
    {
      "commit": "740c77105766feae0f008a67308e6ee4a097e3de",
      "review_id": 4730532027
    }
  ]
}

[github-actions]

Holistic Review

Motivation: The PR targets real, well-documented worst-case inefficiencies in HashSet<T> set operations (intersect/subset/superset/overlap/symmetric-except), backed by convincing benchmarks showing large improvements (e.g. IsSupersetOf 16us -> 1.3us, IntersectWithEnumerable up to ~4x). The motivation is justified.

Approach: The changes are consistent with existing CoreLib collection patterns: a dedicated index-based RemoveAt(entries, removeIndex) (mirroring Dictionary), a Contains(ref Entry) fast path that reuses the precomputed HashCode, BitHelper.FindFirstUnmarked/IsUnmarked/TryMarkBit helpers, firstMarkedForRemove to bound the symmetric-except removal loop, and reference-first comparer equality checks. Value-type vs. reference-type branching preserves JIT devirtualization. The approach is sound and idiomatic.

Summary: ⚠️ Needs Human Review. I verified the subtle correctness-sensitive parts and found no concrete bug, but this is a large rewrite of hot, security/robustness-sensitive CoreLib collection code with no new tests in the PR. A maintainer should confirm the existing System.Collections test suite exercises the added HashSet-vs-HashSet fast paths (Overlaps, IsSupersetOf, effective-comparer mismatches such as randomized string comparers) and the CheckUniqueAndUnfoundElements empty-set path change, and should rely on CI perf/functional coverage.

---

Detailed Findings

✅ RemoveAt free-list bookkeeping is correct

The new index-based RemoveAt correctly unlinks from the bucket chain (both head and mid-chain cases), sets the removed entry's Next to StartOfFreeList - _freeList, clears the value for reference-containing T, and assigns _freeList = removeIndex, _freeCount++. This matches the established Dictionary/HashSet.Remove pattern. Iterating 0..count while calling RemoveAt is safe: RemoveAt only marks the removed entry as free (Next < -1); predecessor entries stay live (Next >= -1), so the entry.Next >= -1 liveness filter never skips a not-yet-visited live entry.

✅ FindFirstUnmarked padding-bit safety

FindFirstUnmarked scans _span for the first word != ~0 and returns i*32 + TrailingZeroCount(~word). When originalCount is not a multiple of 32, the high padding bits of the final word are 0 (unmarked), so this can return an index >= originalCount. Both the enclosing for (; i < originalCount; i++) bound and the inner bitHelper.IsUnmarked(i) re-check guard against acting on padding indices, so the result is correct.

✅ Comparer-equality refactor preserves semantics

EqualityComparersAreEqual and the newly-private EffectiveEqualityComparersAreEqual add a _comparer == _comparer reference fast path before falling back to .Equals. For value types both default comparers are null (equal), and an explicit non-default comparer yields a non-null _comparer, so null vs non-null correctly reports inequality. Removing EffectiveComparer in favor of comparing _comparer directly is consistent with "the actual comparer used to hash entries." The IsSupersetOf/Overlaps fast paths correctly gate on the effective (hashing) comparer before using Contains.

✅ Doc-comment fix

The IsSupersetOf summary was corrected from "proper superset" to "superset" (a pre-existing copy/paste error), and the IntersectWithEnumerable comment _slots -> _entries. Both are accurate.

⚠️ No tests added for reworked behavior

The PR modifies only the two source files. The reworked paths include several new branches worth explicit coverage: HashSet-vs-HashSet Overlaps/IsSupersetOf fast paths, the effective-comparer-mismatch fallback inside IsSubsetOfHashSetWithSameComparer/IntersectWithHashSetWithSameComparer (e.g. one set using a randomized string comparer after hash-collision promotion), and the removal of the _count == 0 special case in CheckUniqueAndUnfoundElements. These are almost certainly covered by the existing extensive System.Collections HashSet tests, but a maintainer should confirm rather than assume, given the correctness sensitivity of this code.

💡 stackalloc inside conditional expression

The refactor inlines stackalloc int[StackAllocThreshold].Slice(...) directly into the ternary that also has a heap-allocation branch. This is valid (the stackalloc only executes on the chosen branch) and matches the intent, but is slightly less obvious than the prior explicit Span<int> local. Purely stylistic; no change required.

Note

This review was generated by this repository's Holistic Review agentic workflow to complement the built-in Copilot review.

Generated by Holistic Review · 172.6 AIC · ⌖ 11.1 AIC · ⊞ 10K