Count differing bits a vector at a time for single-byte elements

[tahakocal]

TensorPrimitives.HammingBitDistance is fully scalar:

long count = 0;
for (int i = 0; i < x.Length; i++)
{
    count += long.CreateTruncating(T.PopCount(x[i] ^ y[i]));
}

For byte/sbyte spans — which is what bitwise Hamming distance is usually computed over, since the inputs are binary fingerprints or quantized embeddings — that is one population count instruction per byte.

Change

Single-byte spans now go through a vectorized path: exclusive-or a vector from each input, look up the bit counts of the low and high nibbles in a 16-entry table with Vector128.Shuffle, and accumulate into byte lanes. A byte lane takes at most 8 per vector, so it is reduced to the running total every 31 vectors, before it can overflow. The tail stays scalar.

Wider element types deliberately keep the scalar loop, and this is the part I would most like reviewed: a 4-byte element already amortizes one population count over 4 bytes and an 8-byte element over 8, and when I measured the same vectorized shape for them it was slower than the scalar loop — 0.63x for int, 0.32x for long. I also measured reinterpreting the spans as 64-bit words and using BitOperations.PopCount on those, which was slower still. So the change is scoped to the case where the measurement supports it.

Vector256/Vector512 are not used: their byte Shuffle is a full cross-lane shuffle rather than the per-128-bit-lane vpshufb, which would undo the gain on x64.

Measurements

Standalone harness (I could not build the runtime on this machine), Apple M-series arm64, Vector128, minimum of 7 rounds, ns per call:

element	length	scalar	vectorized	speedup
byte	1,024	240.3	91.4	2.63x
byte	8,192	1,874.2	724.8	2.59x
byte	65,536	14,964.9	5,895.4	2.54x
byte	1,000,000	234,352	91,240	2.57x
byte	128	33.7	29.7	1.13x

For reference, the same shape applied to wider elements (not taken in this change): int 0.63x, long 0.32x.

Correctness

Differential against the scalar implementation in the harness: 90,000 cases over byte, int and long, lengths 1–140, random values — 0 mismatches, including every length that leaves a partial vector and lengths below one vector where the new path is not taken.

I have not run the System.Numerics.Tensors test suite locally for the reason above, so CI is the first full validation.

[azure-pipelines]

Azure Pipelines:
Successfully started running 3 pipeline(s).
13 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-numerics
See info in area-owners.md if you want to be subscribed.

[tahakocal]

Follow-up on the last line of the description: I have now built the runtime locally. System.Numerics.Tensors.Tests on osx-arm64 Release from this branch: 5,644 total, 0 errors, 0 failed, 0 skipped.