feat: report operators that fall back to PyTorch - #4710
Open
shoumikhin wants to merge 2 commits into
Open
Conversation
narendasan
reviewed
Sep 10, 2026
| self.unsupported_operators: Dict[str, int] = {} | ||
| # unsupported_operators skips impure nodes, so it cannot answer "did anything | ||
| # fall back". This one records every refusal. | ||
| self.fallback_operators: Dict[str, int] = {} |
Collaborator
There was a problem hiding this comment.
Might be nice to include some metadata about reasons for fallback?
| # asked for. The set can hold either a qualified name string or an operator target | ||
| # object, and fallback_operators is keyed by name, so normalize to names first or a | ||
| # target object never matches and the caller is warned about their own choice. | ||
| excluded_names = { |
Collaborator
There was a problem hiding this comment.
I think its fine to report both if we include reasons
| f"{node_name} + Operator Count: {count}" | ||
| for node_name, count in sorted(reported_fallbacks.items()) | ||
| ) | ||
| logger.warning( |
Collaborator
There was a problem hiding this comment.
Should this be warning since its expected behavior? Info might be better so it doesnt need to be suppressed
shoumikhin
force-pushed
the
upstream/fallback-summary
branch
from
September 11, 2026 04:14
fdc87bf to
fd0037e
Compare
## Problem A model can split into TensorRT engines and PyTorch sections without naming the unsupported operators at the default logging level. Users may only notice the split when performance changes. The existing support record also omits refused operations with side effects, such as random number generation. ## Change Record every refused executable operation in both the fast and global partitioners. Use that record for the dry-run report and one warning from the shared compilation path. Name the operators that caused fallback, but leave out operations the caller explicitly chose to run in PyTorch through `torch_executed_ops`. Fully supported models remain silent. The warning covers graphs that reach partitioning. Small graphs that return earlier keep their existing skipped-compilation message. ## Tests Passed 7/7 focused tests. Three fail without this change because the warning is missing: an unsupported operation with side effects, an unsupported complex data type, and fallback through the global partitioner. The other four tests check that supported models stay silent under both partitioners and that caller-requested fallback stays silent when specified by an operator or its name. Those controls pass before and after. The tests build engines and inspect the graph sections and logs. They do not execute the returned models to compare outputs. Tested on Linux x86_64 with Python 3.12 and TensorRT 11.2. Windows, aarch64, TensorRT-RTX, concurrent logging, nested hierarchical graphs, and downstream log parsers were not tested.
Fallback is expected behavior, but a warning without a reason makes it hard to distinguish caller choices from unsupported operations. Include observed reasons and requested fallback in the summary, and report it at INFO without changing support decisions. Test Plan: Passed 10 support tests and 12 compilation/logging tests on Linux x86_64 with CUDA. Both partitioners report reasons and requested fallback; fully supported graphs stay silent and WARNING suppresses the summary. Four rank-boundary execution tests also passed with PyTorch output comparisons. The support tests fail without the reason metadata.
shoumikhin
force-pushed
the
upstream/fallback-summary
branch
from
September 12, 2026 04:58
fd0037e to
9beb4ed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A model can split into TensorRT engines and PyTorch sections without a concise explanation of which operations stayed in PyTorch. The existing support record also omits refused operations with side effects, such as random number generation.
Change
Record every refused executable operation in both the fast and global partitioners. Use the counts for the dry-run report and emit one summary at
INFOfrom the shared compilation path.The summary names each operation, its count, and the reasons observed during support checks. Reasons cover explicit device transfers, tensor rank limits, complex data types, data-dependent output shapes, caller exclusions through
torch_executed_ops, and the lack of a converter that passes validation. A registered converter that rejects a node is not described as missing.Caller-requested fallback is included. Fully supported models remain silent, and setting the log level to
WARNINGhides the summary. Support decisions do not change.The summary covers graphs that reach partitioning. Small graphs that return earlier keep their existing skipped-compilation message. It does not enumerate supported operations left in PyTorch because a partition is too small.
Tests
Passed 10 support tests and 12 compilation/logging tests. They cover both partitioners, refusal reasons, caller exclusions by name and target, mixed fallback counts, INFO severity, WARNING-level suppression, and silence for fully supported graphs. Four logging controls fail on the base branch because the summary is absent. The first commit's seven tests also passed independently.
The reporting tests compile models and inspect logs; they do not execute the returned models. Four existing rank-boundary tests separately passed compiled execution and comparisons with PyTorch.
The rerun used Linux x86_64, CUDA, Python 3.12, PyTorch 2.15 nightly, and TensorRT 11.3 with the Python runtime. Earlier coverage used TensorRT 11.2 with the native runtime. A native TensorRT 11.3 build, Windows, aarch64, and TensorRT-RTX were not tested. Formatting, import-order, and lint checks passed.