Ship only the TensorRT delegate in the ExecuTorch runtime wheel - #4567
Open
shoumikhin wants to merge 78 commits into
Open
shoumikhin wants to merge 78 commits into
shoumikhin wants to merge 78 commits into
Conversation
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
3 times, most recently
from
August 23, 2026 19:00
44796ff to
3c104cb
Compare
shoumikhin
marked this pull request as ready for review
August 23, 2026 19:08
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
from
August 23, 2026 19:27
3c104cb to
4adc20b
Compare
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
from
August 23, 2026 19:30
4adc20b to
7cac1af
Compare
shoumikhin
marked this pull request as draft
August 23, 2026 19:31
shoumikhin
marked this pull request as ready for review
August 23, 2026 19:57
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
7 times, most recently
from
August 24, 2026 17:00
ff3379e to
a008221
Compare
shoumikhin
force-pushed
the
executorch-slim-runtime-wheel
branch
4 times, most recently
from
August 25, 2026 06:55
49b052e to
370c368
Compare
The step that builds the companion wheel was gated on the row's CUDA major, because the release matrix still carried a CUDA 12.6 row and that channel has no ExecuTorch build, so the step's own install failed and took the whole job with it. The CUDA 12 rows are gone. Every matrix that reaches this step now offers CUDA 13 only, and no caller turns on the Jetson path that kept a CUDA 12.6 entry. So the condition can never be false, and a condition that can never be false is worse than no condition: it states a rule the code no longer enforces, and it repeats a value the matrix filter already derives. Test plan: Ran the ExecuTorch test directory: 248 passed. Confirmed no caller enables the Jetson path, and that every wheel and release matrix offers CUDA 13 rows only.
A scheduled workflow run reaches only the default branch, so the arm that gave a release branch the stable track on a schedule could never execute. Worse, the check that stops a shipped release from being re-pinned lived inside that arm, so the one guard that mattered was unreachable while a manual dispatch, the only way a release branch is ever re-pinned, walked straight past it. The dispatch path now carries that check. It also matches a final tag rather than any tag beginning with the version, because a release candidate is not a release and a dispatch during one should still be allowed to move the pin. The header comment already described the real behaviour. It now also says why, so the next reader does not restore the arm. Test plan: Ran the ExecuTorch test directory: 252 passed. Sampled published workflow runs for the schedule event and found every one on the default branch, which is what makes the removed arm unreachable. Confirmed a dispatch on a tagged release refuses, a dispatch on an open release still takes the stable track, a dispatch asking for a nightly off the default branch still refuses, and a schedule on any other branch still does nothing.
The ExecuTorch extra carries a Linux marker, so off Linux the suggested pip command matches nothing, installs nothing, and still exits successfully. A user on Windows or macOS with a CUDA 13 build followed it, saw it succeed, and hit the same error again. This is reachable rather than theoretical, because the project builds Windows CUDA 13 wheels. The helper now checks the platform as well as the CUDA version, and falls back to the same guidance it already gives an unsupported CUDA build, which names the requirement. The load error for a missing delegate runtime also claimed the wheel is not published to any index, which stopped being true once it started shipping. It now names where to get it. Test plan: Ran the ExecuTorch test directory: 252 passed. Confirmed the helper returns an install command on Linux and the guidance text on Windows and macOS, with the same supported CUDA build in all three cases, so only the platform decides. Confirmed the wheel the load error described as unpublished is present on the index the new message points at.
Writing happens per file after every site has been read and rewritten in memory, so a parsing problem cannot split the tree. A disk or signal failure partway through the writes still can, and the tool then refused every later run, including one aiming at a newer version, because the sites it had already moved no longer matched the version it expected to replace. A site already carrying the target now counts as satisfied, so the next run completes what the interrupted one started instead of needing a human to unpick it. The docstring claimed everything was preflighted before any write. That was only ever true of parsing, so it now says what the function actually guarantees. Test plan: Ran the ExecuTorch test directory: 252 passed. Confirmed a run whose sites are split between the old and the target version succeeds and converges, and that the same case fails without this change.
Bundling a second ExecuTorch runtime makes backend registration depend on import order. Ship one delegate linked to the installed runtime, with import-time registration and direct Module API examples that preserve device-resident inputs. Keep companion builds, repair, artifact checks and uploads on the shared Linux path. Build the companion for CUDA 13.0 and 13.2, retain its independent version, and leave ordinary release rows unchanged. This removes the custom runtime wrapper and the released loader entry point. The API removal still requires a compatibility path or an explicit maintainer exception before merge. Test Plan: 142 CPU-safe runtime checks passed, including real wheel tagging, setup metadata, CMake version checks, native-guard fixtures and example API routing. All 178 pin/updater checks passed, including installed-wheel provenance and the history-free repository integration. Downstream native operations were stubbed in CPU-safe checks. No fresh native Linux/CUDA build or delegate GPU execution was run.
A preloaded delegate can lose registration to another library. Accepting it by name alone reports success while the other library remains active. Check ownership before retaining the loaded handle, while preserving valid preload adoption and repeated registration. Test Plan: 40 loader tests passed, including two presence-only removal controls; 40 passed again from the exported staged tree. The combined CPU-safe packaging and loader suite passed 154 tests, and the pin/updater suite passed 186. Six native load-order and concurrency cases passed on Linux aarch64. A native presence-only control reproduced the wrong acceptance, and both missing-query paths were rejected. Full wheel and numerical GPU validation remain separate.
Repeated package discovery must restore the caller's library list even when the imported target already exists. Select the fixed delegate filename shipped by the wheel so neighboring versioned files cannot replace it. Test Plan: Six configure-only discovery cases and four guard-removal controls passed, including repeated and function-scoped calls. The same ten checks passed from the exported staged tree. A broader selection of 163 CPU-safe packaging and loader tests passed.
Consumers must get RUNPATH even when their toolchain defaults to old dynamic tags. Preserve scoped delegate retention so static registration runs without keeping unrelated libraries. Test Plan: A Linux native fixture reproduced the old-dtags failure before repair. Six fixture cases passed afterward and from the staged snapshot, including retention, ordering, scope and new-dtags removal controls. The real ARM64 delegate consumer retained its registration and emitted RUNPATH. Thirteen existing CMake checks passed.
A consumer requesting a range that spans major versions must not silently accept the companion. Match the standard same-major rules while preserving the exclusive next-major boundary and independent companion version. Test Plan: Eleven real CMake package requests agree with the standard generator. Three guard-removal controls reproduce the original cross-major acceptance. All fourteen cases pass from the staged snapshot. The selected CPU-safe suite passed 176 cases, and the complete pin/updater suite passed 186 cases.
Requesting a CMake component does not link it. Link the CUDA backend explicitly so both documented mixed consumers register CUDA and TensorRT. Tested ten CPU-native consumer cases and four installed-runtime registration cases on Linux aarch64, including CUDA-link removal controls. Repeated the repository cases from the staged snapshot.
Reject a model path ending in .expected before exporting or writing the reference. Separate model and reference paths retain the existing naming. Tested four output-boundary cases, including existing-file preservation and a removed-guard control, from the working tree and an independent staged snapshot.
Reject a non-CUDA input explicitly before loading a device-resident program. Python optimization no longer removes the guard. Tested normal and optimized compilation with invalid inputs, valid inputs, and guard removal. All ten example-boundary cases passed from an independent staged snapshot.
Keep format="executorch" working through the installed ExecuTorch Module loader. Preserve the released method interface and CPU input behavior, and warn callers to migrate during a minimum six-month deprecation period. Tested 82 compatibility, runtime-policy, and example-boundary cases from the exact staged snapshot. Nine removal controls detected broken behavior. Real TensorRT and CUDA execution are not covered by these tests.
Require an explicit true value before allowing an unpinned ExecuTorch build or selecting debug mode. Nonempty false values must not change build safety or optimization. Test Plan: Executed actual setup commands for 30 flag cases and two removed-check controls. The exact staged snapshot passed 72 packaging and Python runtime cases.
Use the same case-insensitive true values as the build flags so tooling can consistently suppress import-time registration. Test Plan: Executed imports for 15 flag values and a missing-on control. The exact staged snapshot passed 88 packaging and Python runtime cases.
Editable installs lost the native library and CMake files when temporary build directories were removed. Build them beside the source package and report the stable output mapping to setuptools. Keep ordinary wheel payloads and build-directory cleanup unchanged while preserving source-root files. Test Plan: 98 packaging and runtime cases passed from the staged snapshot. Real default and strict editable builds and pip installs load a compiled probe after temporary directories disappear; removing the fix fails both modes. Ordinary wheel members are byte-identical with the fix removed. All 42 packaging cases also pass on setuptools 68.0.0 and wheel 0.40.0, and 16 existing version cases pass. These are controlled packaging tests, not a CUDA delegate build.
The test workflow installs the companion wheel. Source loading is deliberate so these tests exercise checkout code with controlled dependencies, not evidence that the wheel is absent. Test Plan: Verified all four misleading comments were replaced and executable syntax is unchanged. All 98 packaging and runtime tests passed from the exact staged snapshot.
The main wheel is installed without dependencies, so a clean builder lacks the TensorRT Python metadata required by companion setup. Install only the TensorRT requirements declared by that wheel, preserving the matrix's torch and architecture-specific dependency selection. Test plan: execute the shared build commands with metadata fixtures for both architectures, CUDA 13.0/13.2, and release/nightly versioning. Both dependency installation removal controls fail as expected. Selected workflow checks pass. No native build or GPU execution is claimed.
Source-text assertions accepted commented or disabled device commands. Run the parsed workflow script with bounded fake tools and record export and runner arguments. Check artifact handoff, failure propagation, and exclusion of device inputs from the host-input reference runner. Keep serialized-program checks separate. Correct the builder GPU and registry-only coverage descriptions and remove the stale line citation. Test plan: CPU-only workflow execution, both command failure controls, commented/disabled/removed command controls, and a disabled-job control. The selected workflow and wheel-policy suite passes. No numerical GPU execution is claimed.
The unpinned fixture used an exact wrong version and omitted unrelated required dependencies. Use a genuine range while retaining valid dependency metadata, and keep the separate exact-wrong-pin case. Test plan: the real checker rejects the range through its version branch. Deleting only exact-version enforcement admits that same wheel. The original fixture still failed on a missing dependency under this control.
Run the shared installed-check commands through bounded tools using real wheel metadata and archive extraction on both architecture tags. Keep native loading simulated, but require the installed path and clear build-only library paths. Bound the existing shared repair test as well. Test plan: successful checks and missing-library, undefined-symbol, ldd, and import failures; deleted/false resolution guard and disabled-step controls. Real wheel tags and RECORD hashes survive shared repair. No native ELF or GPU execution is claimed.
Check CUDA dependencies against the artifact's actual RUNPATH so a valid build does not depend on optional expected text. Reject invalid supplied options and unsupported platform tags instead of silently skipping checks. Use auditwheel 6.8.2 numeric and named symbol policies for the two supported architectures, and propagate dependency-inspection failures. Keep shared registry and CUDA stream linkage, exercise the production post-build check and remove the unused private-runtime guard. Test Plan: 100 artifact cases passed on Linux aarch64; 89 passed with 11 Linux-only skips on macOS. Real GNU, LLVM and elfutils readers accepted valid ELF and rejected legacy RPATH. Production native builds exercised removed, early-return and inert guard controls, alias retention and static-target rejection. Original selected guard definitions passed 43 Linux cases and 42 macOS cases with one Linux-only skip. Complete pin/updater suites passed 186 cases. The initial before-fix run had 41 failures and 36 passes. Full wheel and GPU numerical tests were not run.
Explain rebuilding generated native files, restarting consumers, and explicit boolean flag values. Verified the build-recipe check against the staged documentation.
The companion includes a CMake package for installed consumers. Remove an outdated explanation that described a library-only archive while keeping the existing target checks. Test Plan: Both focused CMake export and installed-package tests passed; assertions are unchanged.
The active build environment can retain an older wheel tool when build isolation is disabled. Require the existing minimum version so companion repair can retag the wheel before artifact upload. Test Plan: 208 pin, updater, and workflow tests passed. Removing the minimum is detected by both architecture cases. Two real wheel archive retagging tests passed with wheel 0.40.0; their payloads are fixtures, not native builds.
Keep controlled packaging and registration checks in the unit selection so they run when the unit marker is requested. Test Plan: Normal API and Python runtime unit selection passed 231 tests on Linux with a compatible published compiler wheel. This checks collection and test behavior, not a final native wheel build.
C++ applications following the installed-wheel example cannot execute programs that copy tensors between the host and GPU because the copy kernels are not registered. Include the required kernel library and document caller-owned outputs for device-resident programs. Extend the native example checks to verify registration and library retention. Test Plan: - Built and ran an installed-wheel C++ consumer on Linux aarch64 with CUDA 13.2. Nine numerical calls passed across TensorRT-only, mixed, and device-resident programs. Omitting the delegate failed with an unregistered backend. - The updated documentation checks failed in six cases before the fix. All 12 native CMake checks passed on the exact staged snapshot afterward.
Fresh wheel-checker subprocesses cannot use the private copy of wheel exposed by setuptools in the parent test process. Install wheel>=0.40 in both ExecuTorch test entry points so artifact checks can run reliably. Tests: 267 pin, updater and workflow cases passed; 42 Linux artifact cases passed, including rejection and removed-guard controls. All 26 workflow cases passed from the final staged tree. Five targeted cases failed before the setup fix.
The delegate tests restated which CUDA rows the shared build accepts, as a literal set in three assertions. The matrix filter already declares that, so the same fact lived in two places and a CUDA version bump had to be applied to both. Read the rows from the filter script instead. The assertions now check that the tests and the filter agree, rather than checking the tests against themselves. Also translate the shared build gate's startsWith expression when evaluating it, so the gate test exercises the expression the workflow actually uses. Test plan: Ran the delegate test module on Linux aarch64. Confirmed the row assertions fail when the filter and the tests disagree.
The companion wheel on the nightly index today exports `activate` and `get_runtime` and ships a `runtime` submodule. This change removed all three with no alias, so code written against the published wheel breaks on upgrade. `activate()` is back as a deprecated alias for `register()`. `get_runtime()` raises with what to do instead, since this package no longer bundles a runtime and there is no runtime object left to return. The loader had the same break from the other side: it imported `register` unconditionally, which raises ImportError against that published companion. ImportError is not ModuleNotFoundError, so its own handler did not catch it and the failure surfaced as a traceback instead of the guidance the function exists to give. It accepts either entry point now, so upgrading the main wheel alone keeps loading programs during the compatibility window this change promises. Also make a failed native build fail the install. setuptools runs a customized `build_py` inside a broad `except Exception` during editable installs and downgrades the error to a warning pip hides without `-v`, so a failed delegate build left pip printing "Successfully installed" with no delegate present. SystemExit escapes that catch. A misconfiguration this file raises itself is left as it is, so the reason a build stopped is still the reason it reports. The delegate lane's matrix flag was asserted by nothing, so deleting it left every guard green while the lane tested rows whose channel carries no ExecuTorch. Test plan: Ran the pin, updater and shared runtime suites on Linux aarch64: 287 passed. Downloaded the companion published on the nightly index and confirmed it exports activate and get_runtime and ships runtime.py, then confirmed the loader resolves that wheel through activate where it previously raised ImportError. Confirmed the shims are exported, that get_runtime explains itself rather than failing obscurely, and that the filter keeps only CUDA 13 rows when the lane passes its flag. Deleting the flag from the workflow fails that test.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The runtime wheel bundled a second copy of the ExecuTorch runtime and swapped in its own Python
bindings at import time. Two copies of a runtime in one process can disagree, and whether loading
worked depended on import order. The TensorRT delegate should use the ExecuTorch runtime the user
already installed.
Change
Ship the TensorRT delegate library, its Python registration module, and a CMake package for C++
consumers. Build against the installed ExecuTorch wheel rather than bundling a private runtime, so
there is one runtime in the process.
Keep the delegate on the shared build, repair, artifact and upload path the main wheel already uses,
so its packaging cannot drift from the wheel it ships beside. It keeps its own version, and it pins
the exact ExecuTorch it was built against, because the delegate links that runtime's shared library
and the two have to be the same build.
Take the TensorRT requirement from the main wheel's own declared dependencies instead of repeating
it, so a delegate build cannot select a different TensorRT than the wheel it links.
Keep
torch_tensorrt.load(..., format="executorch")working, with the entry points the publishedpackage already exposes, so upgrading this wheel alone does not break existing callers. Examples use
ExecuTorch's own Module API. The C++ guidance covers linking the CUDA backend alongside this one and
explains caller-owned outputs for device-resident exports.
Test plan
Ran the ExecuTorch test directory on Linux.
Built and installed the main and companion wheels on Linux aarch64 with CUDA 13.2, then exported and
numerically executed four kinds of program through stock ExecuTorch: TensorRT only, mixed CUDA and
TensorRT in one program, device resident, and externally weighted. Ran the same programs from a C++
consumer built against the installed wheels.
Confirmed the controls fail as expected: a missing delegate and missing external data both error
rather than silently producing wrong output.
Checked the shipped library's dependencies, symbol retention and library search paths, and that a
C++ consumer resolves CUDA, TensorRT and ExecuTorch from the installed wheels.
The x86_64 main wheel passed installation and dependency checks. Full x86_64 companion qualification
is not complete, and one Arm platform-tag check is still open.