feat(ci): add Linux x86_64 + aarch64 CUDA release builds with bundled CCCL - #262
Merged
Conversation
Part of #25. Add a build-linux-x86_64-cuda job to the release workflow that produces mlxcel-linux-x86_64-cuda13.zip from the self-hosted x86_64 NVIDIA runner (labels: self-hosted/Linux/X64/CUDA). - Fat binary over Ampere..Blackwell: MLX_CUDA_ARCHITECTURES=80;86;89;90a;100;120. 90a is load-bearing for MLX's Hopper quantized kernel gate, and build.rs forwards the list verbatim. sm_87 (Jetson) and sm_121 (GB10) are aarch64-only. - GPU smoke test: download Qwen3-0.6B-4bit and generate 10 tokens before packaging, so a broken binary fails the job (issue #25 acceptance criterion). - Hoist MLX_EXPECTED_COMMIT to a workflow-scoped env shared by both CUDA jobs, removing the duplicated pin literal. - Add a linux-x86_64 dispatch target so this job can be built in isolation. - Honor the documented "empty release_tag = no upload" contract across all jobs: upload only on a real release event or a dispatch with a non-empty tag.
MLX's CMake resolves cblas.h and lapacke.h; liblapack-dev alone omits the lapacke header and the configure step fails with LAPACK_INCLUDE_DIRS set to NOTFOUND. List the concrete Debian/Ubuntu build packages and note that the release workflow now also produces a single Linux x86_64 CUDA artifact fattened over Ampere..Blackwell, alongside the per-target aarch64 binaries.
Closed
5 tasks
MLX's CUDA backend NVRTC-JITs gather/indexing kernels at runtime; those include <cuda/std/tuple>, so the CCCL (libcu++) headers must be present on the deployment host. The compiled-in MLX_CCCL_DIR points at the build machine and does not exist elsewhere, so a flat binary aborts on the first generate with "cannot open source file cuda/std/tuple" on any host other than the builder (the build-box smoke test masked this). Bundle CCCL in MLX's native runtime layout for both Linux CUDA jobs (aarch64 and x86_64): package as bin/ + include/cccl/, which is where the JIT looks (<exe-dir>/../include/cccl). Only cuda/ and nv/ are shipped (~8.8 MB). A sanity check asserts cuda/std/tuple is present before zipping. Host CUDA runtime headers still come from the node's CUDA_HOME. Document the prebuilt-artifact runtime requirements (layout, CUDA_HOME, MLX_PTX_CACHE_DIR) in docs/installation.md. A follow-up (#265) will add an MLXCEL_CCCL_DIR override and binary-adjacent lookup for flat-layout embedding.
The package step piped `unzip -l | awk | head -20`; head closed the pipe early, SIGPIPE killed unzip, and set -o pipefail turned that into exit 141, failing an otherwise-correct packaging step (the zip and CCCL bundle were already built). Drop head and let awk read the whole stream and self-limit its output. Applies to both Linux CUDA jobs.
Ship the mlxcel CLI and mlxcel-server as separate per-target archives (each ~347 MB) instead of one combined zip, so a consumer pulls only what it needs. Both archives carry the CCCL headers in the bin/ + include/cccl/ runtime layout. Names: mlxcel-<target>.zip and mlxcel-server-<target>.zip. Collapse the aarch64 matrix (gb10=121, gh200=90a) into a single job that builds one fat binary covering both NVIDIA aarch64 targets (90a;121), cross-compiled on the GB10 runner, mirroring the x86_64 fat build. Add a linux-aarch64 dispatch target so it can be built in isolation. Apply the same packaging to both Linux CUDA jobs and update docs/installation.md.
…meouts The aarch64 archive now covers GH200 (90a), GB200 (100), and GB10 (121) in one build (90a;100;121). sm_100 already compiles in MLX (proven by the x86_64 build). Raise timeouts preemptively for the larger cold builds: aarch64 180->360, x86_64 300->480 (a cold 6-arch x86_64 build was observed at ~3h).
…inary MLX resolves the bundled CCCL headers from the executable's own path via dladdr; a relative `./mlxcel` from inside bin/ yields a relative dli_fname and the include/cccl lookup misses, failing the first NVRTC JIT. Launch via the absolute path (as a service manager / subprocess spawn does). Tracked for a proper MLXCEL_CCCL_DIR override.
inureyes
added a commit
that referenced
this pull request
Jun 13, 2026
…ent kernel cache (#270) ## Summary Three CUDA NVRTC-JIT UX/robustness fixes, all in or around the bundled MLX `jit_module.cpp` overlay. Closes #265 (the first item); the other two were folded in since they touch the same JIT path. ## 1. Robust bundled-CCCL resolution (Closes #265) PR #262 bundles the CCCL (libcu++) headers in each Linux CUDA archive (`bin/` + `include/cccl/`). But MLX resolved that location via `current_binary_dir()`, which uses `dladdr`; a relative launch (`./mlxcel` from inside `bin/`) yields a relative `dli_fname`, so the lookup missed and the first gather/indexing kernel JIT aborted with `cannot open source file "cuda/std/tuple"`. Confirmed on a Backend.AI GPU node: absolute-path launch worked, relative did not. Patch `include_path_args()` to honor an `MLXCEL_CCCL_DIR` env override first, then resolve the executable dir from `/proc/self/exe` (canonical absolute) on Linux, falling back to `current_binary_dir()`. Resolution is now invocation-independent. ## 2. First-run JIT notice The first CUDA generate on a cold cache JIT-compiles gather/indexing kernels via NVRTC, making the first response noticeably slower with no indication. Print a one-time stderr notice at the actual compile site (`mlxcel run` and `mlxcel-server` both surface it). Suppress with `MLXCEL_QUIET_JIT`. ## 3. Persistent, MLX-pin-scoped PTX cache MLX's default PTX cache is the system temp dir (e.g. `/tmp/mlx/0.32.0/ptx`), cleared on reboot, so the first-run compile is paid every boot. Default `MLX_PTX_CACHE_DIR` (when unset) to `${MLXCEL_CACHE_DIR:-~/.cache/mlxcel}/cuda-ptx/<mlx-commit>`, paid once per machine. Scoped by the pinned MLX commit because `read_cached_ptx` is keyed only by kernel name and never validates the source, so entries must not survive an MLX bump. `mlxcel-core` build.rs exposes the pin via `MLXCEL_MLX_COMMIT`; both binaries call `ensure_persistent_ptx_cache()` at startup. Override with `MLX_PTX_CACHE_DIR`. ## Docs `docs/installation.md`: drop the absolute-path caveat, document `MLXCEL_CCCL_DIR`, `MLX_PTX_CACHE_DIR`, and `MLXCEL_QUIET_JIT`. ## Validation - The `jit_module.cpp` overlay is the a6ec712 (pinned MLX) file with only `include_path_args()` and the compile-site notice modified; verified byte-identical to upstream before editing. - The Rust helper compiles in edition 2021 without `unsafe` (set_var is edition-2024-gated). - CUDA build + smoke test: passing (CCCL resolution + notice compiled; an earlier revision already built green). Re-verified after the cache change. - Functional relative-launch fix is proven on a GPU node without the build-machine `MLX_CCCL_DIR` (the build runner's fallback masks it in CI). Closes #265.
inureyes
added a commit
that referenced
this pull request
Jun 16, 2026
…o the existing x86_64 job main (#262) already ships a self-hosted Linux x86_64 CUDA release job (build-linux-x86_64-cuda: one fat binary over 80;86;89;90a;100;120, separate mlxcel / mlxcel-server archives, bundled CCCL headers, GPU smoke test), which supersedes this branch's GitHub-hosted per-arch matrix job. Conflict resolution: - Drop the duplicate build-linux-x86-cuda matrix job. - Keep the MLXCEL_CXX_MARCH build.rs override and apply MLXCEL_CXX_MARCH=x86-64-v3 plus RUSTFLAGS=-C target-cpu=x86-64-v3 to the existing build-linux-x86_64-cuda build step, so its redistributable asset no longer inherits the build host's -march=native ISA (latent SIGILL on older customer CPUs). - Keep notify-teams / promote-release needs on build-linux-x86_64-cuda. - Merge docs: keep main's CCCL runtime-requirements section; add the MLXCEL_CXX_MARCH reference, env-var rows, and the qmm OOM / liblapacke-dev troubleshooting entries. Verified on aarch64 (GB10): cargo build --release succeeds and the build.rs match honours unset->native, none->omitted, supported->passed, unsupported->gracefully dropped.
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add Linux CUDA release builds (issue #25 part A). Both targets are fat binaries, and the
mlxcelCLI andmlxcel-servership as separate, self-contained archives.80;86;89;90a;100;120), with a GPU smoke test (load + 10-token generate).90a;100;121).CCCL bundling
MLX's CUDA backend NVRTC-JITs gather/indexing kernels at runtime, which need the CCCL (libcu++) headers (
cuda/std/tuple). The compiled-inMLX_CCCL_DIRpoints at the build machine, so a flat binary aborts elsewhere withcannot open source file "cuda/std/tuple". Each archive bundles the CCCL headers (~8.8 MB, 953 files) in MLX's native runtime layout:bin/<binary>+include/cccl/.Known limitation: MLX resolves the bundle from the executable's own path via
dladdr, so launch the binary by its absolute path (as a service manager or subprocess spawn does); a relative./mlxcelfrom insidebin/can miss the bundle. A follow-up (#265) adds anMLXCEL_CCCL_DIRoverride and/proc/self/exeresolution to remove this constraint.Other changes
MLX_EXPECTED_COMMITto a workflow-scoped env shared by both CUDA jobs.linux-aarch64/linux-x86_64dispatch targets for isolated builds; raise build timeouts (aarch64 360, x86_64 480).docs/installation.md: requireliblapacke-dev; document the prebuilt-artifact runtime requirements (layout,CUDA_HOME, absolute-path launch,MLX_PTX_CACHE_DIR).Validation
Part of #25.