build: add MLXCEL_CXX_MARCH override and pin the x86_64 CUDA release asset to x86-64-v3

[achimnol]

Summary

Pins the Linux x86_64 CUDA release asset to a portable ISA baseline so it runs on any AVX2-capable CPU, not only the build host. Adds an MLXCEL_CXX_MARCH build-script override and applies it (with a matching RUSTFLAGS target-cpu) to the existing build-linux-x86_64-cuda release job.

Scope note. An earlier revision of this PR added a separate GitHub-hosted, per-architecture matrix job. main (#262) has since landed the x86_64 CUDA release on the self-hosted RTX runner: one fat binary over 80;86;89;90a;100;120, separate mlxcel / mlxcel-server archives, bundled CCCL headers, and a GPU smoke test. To avoid a duplicate (and CCCL-less) job, this PR now keeps only the portable-ISA fix and folds it into that existing job. See the inline comment for the full conflict-resolution notes.

Why

build-linux-x86_64-cuda produces a redistributable asset, but build.rs defaults the C++ bridge to -march=native. On the build runner that bakes the host CPU's ISA (possibly AVX-512) into the binary, which then SIGILLs on older customer CPUs. Pinning a portable baseline for redistributable builds fixes that, while local/dev builds and the per-machine GB10/GH200 assets stay on native.

Changes

• MLXCEL_CXX_MARCH override (src/lib/mlxcel-core/build.rs): release builds read MLXCEL_CXX_MARCH. Unset keeps -march=native (default unchanged); none omits the flag; any other value is passed as -march=<value> via flag_if_supported, so a value the host compiler does not accept is dropped rather than failing the build.
• release.yml: the existing build-linux-x86_64-cuda build step now sets MLXCEL_CXX_MARCH=x86-64-v3 and RUSTFLAGS=-C target-cpu=x86-64-v3, pinning both the C++ bridge and rustc to the portable x86-64-v3 (AVX2, ~2013+) baseline. No new job; notify-teams / promote-release gating is unchanged.
• Docs: MLXCEL_CXX_MARCH reference in docs/environment-variables.md and docs/installation.md (new "C++ ISA baseline" section plus env-var rows), and troubleshooting entries for the qmm Error 137 OOM and the liblapacke-dev (lapacke.h) build requirement.

Verification

• x86_64 (Ubuntu 26.04, CUDA 13.3, RTX 5090): cargo build --release --features cuda --locked succeeds; mlxcel generate produces tokens on the GPU.
• aarch64 (GB10): cargo build --release of the merged tree succeeds; the build.rs match honours unset -> native, none -> omitted, supported value -> passed, unsupported value -> dropped. cargo fmt --check clean.
• Runtime portability of the x86_64 asset on a non-AVX-512 CPU is verified out-of-band on real hardware (no SIGILL), since CI builds without that hardware.

Out of scope / follow-ups

• CPU-only x86_64 asset (the debian/PPA packaging already covers that path).
• Gating the turbo4 sparse-V Metal-JIT kernel path on Metal availability (pre-existing on all Linux builds).

[inureyes]

Merged current main into this branch and resolved the conflicts. One heads-up: the approach changed, because main moved underneath this branch.

main #262 already landed a Linux x86_64 CUDA release job (build-linux-x86_64-cuda) on the self-hosted RTX runner: one fat binary covering 80;86;89;90a;100;120, separate mlxcel / mlxcel-server archives, bundled CCCL headers (required for MLX's runtime NVRTC JIT), and a GPU smoke test. That supersedes the GitHub-hosted per-arch matrix job this branch added.

Resolution:

• Dropped the duplicate build-linux-x86-cuda matrix job.
• Kept the MLXCEL_CXX_MARCH build.rs override (the load-bearing fix here) and applied MLXCEL_CXX_MARCH=x86-64-v3 + RUSTFLAGS=-C target-cpu=x86-64-v3 to the existing build-linux-x86_64-cuda build step. That job was producing a redistributable asset with the default -march=native, so it inherited the build host's ISA and could SIGILL on older customer CPUs; your override closes that gap.
• Kept notify-teams / promote-release gating on build-linux-x86_64-cuda.
• Merged the docs: kept main's CCCL runtime-requirements section and added the MLXCEL_CXX_MARCH reference, env-var rows, and the qmm OOM / liblapacke-dev troubleshooting.

Net diff vs main is now just: build.rs (MLXCEL_CXX_MARCH), the x86-64-v3 env on the existing x86_64 job, and the docs.

Verified on aarch64 (GB10): cargo build --release succeeds and the build.rs match honours unset->native, none->omitted, supported->passed, unsupported->gracefully dropped. The x86_64 asset's portability on older CPUs is inherently out-of-band here (no x86 hardware), as the PR description notes.

Since the GitHub-hosted matrix job is gone, the PR title/body now describe more than the diff does; happy to retitle/rescope to "the MLXCEL_CXX_MARCH portable-ISA fix for the existing x86_64 release asset" if that reads better.