ORT 1.29.1 Cherry Picks - #32288
Merged
Tianlei Wu (tianleiwu) merged 11 commits intoSep 10, 2026
Merged
ORT 1.29.1 Cherry Picks#32288
Conversation
This pull request improves the robustness of the FastGelu fusion optimization by ensuring malformed nodes are properly skipped and adds a test to verify this behavior. The main changes include stricter input validation in the fusion logic and a new unit test. **Fusion logic improvements:** * Added explicit checks for the number of inputs (`InputDefs().size()`) in `Mul` and `Pow` nodes within the `FastGeluFusion` optimizer to ensure only well-formed nodes are considered for fusion. [[1]](diffhunk://#diff-8f18e5c2ad33a6cc11a340f2c0ff3ce5ad63beed1dcc31ea49f1ff409ef030c9R39) [[2]](diffhunk://#diff-8f18e5c2ad33a6cc11a340f2c0ff3ce5ad63beed1dcc31ea49f1ff409ef030c9L89-R91) [[3]](diffhunk://#diff-8f18e5c2ad33a6cc11a340f2c0ff3ce5ad63beed1dcc31ea49f1ff409ef030c9R119) **Testing enhancements:** * Introduced a new test, `FastGeluFusionSkipsMalformedScaleMul`, that modifies a model to create a malformed `Mul` node and verifies that the fusion optimizer correctly skips it (i.e., does not produce a `FastGelu` node).
This pull request introduces stricter validation and error handling for initializers with in-memory external data references in ONNX Runtime's graph handling. The main goal is to ensure that all such references are properly registered and that their data matches expectations, preventing invalid model states and improving robustness. Additionally, new tests are added to verify these behaviors. **Validation and Error Handling Improvements:** * Added a new `ValidateInMemoryInitializers` method to the `Graph` class, which checks that all in-memory external data initializers have corresponding `OrtValue` objects with matching data, and integrated this validation into the graph transformation process. [[1]](diffhunk://#diff-aaea1507ec81a94c72a1fa72ce320df712156b665f7798573be3f7e439bb4c37R1579-R1583) [[2]](diffhunk://#diff-e231a92b40d89409cc8e82436be0a15bc87ef95c93b303b9feaeab6e50c8835cR4000-R4023) [[3]](diffhunk://#diff-3e2227e1225091e8b74c02688e23b21630d1393dd395e15966558901538dd2c7R1549-R1551) * Introduced a helper function `GetValidatedInMemoryInitializer` in `graph_utils.cc` to enforce that in-memory external data initializers are registered and their data matches, replacing ad-hoc checks in various code paths. * Updated `MakeInitializerCopyIfNotExist` and `ConvertInMemoryDataToInline` to use the new validation helper, ensuring consistent and early detection of invalid initializer states. [[1]](diffhunk://#diff-0791c3ebdddb6f4be85d07b707d494551597574eb4f198b0a476d6602c7e2d8bR495-L496) [[2]](diffhunk://#diff-0791c3ebdddb6f4be85d07b707d494551597574eb4f198b0a476d6602c7e2d8bR530) **Testing Enhancements:** * Added the `RejectsUnregisteredInMemoryInitializerCopy` test to verify that the system correctly rejects initializers with arbitrary or unregistered in-memory references, both during validation and when attempting to copy such initializers.
adrastogi
requested review from
Akshay Sonawane (apsonawane) and
Tianlei Wu (tianleiwu)
August 26, 2026 23:51
Tianlei Wu (tianleiwu)
previously approved these changes
Aug 27, 2026
### Description <!-- Describe your changes. --> Fix Graph::ToGraphProtoInternal to clear the destination GraphProto before populating it, rather than clearing the graph's backing proto. Add regression coverage for Compile API output using both an output-model write callback and a custom initializer-location callback, including: - Models with no initializers. - Embedded initializers. - External initializers. - Reloading the emitted model and running inference. - Verifying inputs, outputs, nodes, and initializers are serialized exactly once. ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> When compilation produced no EPContext nodes, the Compile API emitted a plain optimized ONNX model. If an output write callback and custom initializer-location callback were both configured, serialization appended graph fields to an already-populated destination. This duplicated nodes, inputs, outputs, and value information. CompileModel returned success, but loading the emitted model failed with: Error: Duplicate definition-site for (X). Clearing the destination proto before repopulating it ensures the emitted model remains valid while preserving existing embedded and external initializer handling. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tianlei Wu (tianleiwu)
previously approved these changes
Aug 29, 2026
### Description
GroupQueryAttention previously always applied a causal mask. This adds a
`causal` attribute, defaulting to `1` for backward compatibility.
- **CPU**
- Uses bidirectional masking when `causal=0`.
- Routes bidirectional execution through the compatible unfused path.
- Rejects `local_window_size != -1` with bidirectional attention because
local-window alignment is defined only for causal attention.
- **CUDA**
- Propagates the attribute across Flash Attention, memory-efficient
attention, cuDNN SDPA, and unfused paths.
- Excludes causal-only XQA for bidirectional attention.
- Rejects `local_window_size != -1` with bidirectional attention.
- Quantized bidirectional KV-cache execution requires Flash Attention.
MEA and unfused fallbacks do not consume quantized KV caches and return
`NOT_IMPLEMENTED` instead of reading them incorrectly.
- **Other EPs**
- WebGPU and JS report `NOT_IMPLEMENTED` for `causal=0`.
- DML rejects `causal=0` during kernel creation, and WebNN declines the
node during capability checks, avoiding silent causal output.
- **Coverage**
- Adds default-causal and bidirectional CPU/CUDA mask tests with
identity-sensitive Q/K logits.
- Adds non-quantized and quantized bidirectional past-KV parity
coverage.
- Adds local-window rejection and WebGPU rejection tests.
```cpp
tester.AddAttribute<int64_t>("causal", 0);
```
### Motivation and Context
Bidirectional models require each query token to attend to the full
valid key sequence. The new attribute enables this on CPU and CUDA while
preserving existing causal behavior by default. Generation conversion
stamps `causal=1` explicitly because its decoder attention is
unidirectional by definition.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
## Description Enables the CPU GroupQueryAttention implementation to use `attention_bias` with `sliding_window_cache`. This is needed by speculative decoding with sliding-window attention, including calls that provide explicit `position_ids` for RoPE. ## Summary of Changes ### CPU GQA | File | Change | |------|--------| | `onnxruntime/contrib_ops/cpu/bert/group_query_attention.cc` | Derives the absolute KV origin for each cache-relative batch and forwards it to attention implementations. | | `onnxruntime/contrib_ops/cpu/bert/gqa_attention_base.h` | Applies the absolute bias-column offset in quantized, unquantized, flash, and non-flash paths, with per-batch fallback for differing origins. | | `onnxruntime/contrib_ops/cpu/bert/group_query_attention_helper.h` | Allows windowed attention bias only for callers that explicitly support absolute bias offsets; CUDA and WebGPU behavior is unchanged. | ### Tests - Adds a deterministic CPU regression covering post-eviction bias indexing with explicit `position_ids`. - Extends windowed-cache parity coverage to combine attention bias, non-default explicit position IDs, repeated eviction, and forced non-flash dispatch. ## Testing - `cmake --build build/ci_cpu/Release --target onnxruntime_provider_test -j 8` - `build/ci_cpu/Release/onnxruntime_provider_test --gtest_filter=GroupQueryAttentionTest.WindowedCacheAttentionBiasWithPositionIds_CPU` - `ORT_GQA_DISABLE_FLASH_ATTENTION=1 build/ci_cpu/Release/onnxruntime_provider_test --gtest_filter=GroupQueryAttentionTest.WindowedCacheAttentionBiasWithPositionIds_CPU` - `clang-format --dry-run --Werror onnxruntime/contrib_ops/cpu/bert/group_query_attention_helper.h onnxruntime/contrib_ops/cpu/bert/gqa_attention_base.h onnxruntime/contrib_ops/cpu/bert/group_query_attention.cc onnxruntime/test/contrib_ops/group_query_attention_op_test.cc` - `python3 -m py_compile onnxruntime/test/python/transformers/test_gqa.py` ## Motivation and Context A windowed KV cache stores resident rows in cache-relative coordinates after eviction, while `attention_bias` remains indexed by absolute sequence position. The previous validation rejected the combination to avoid silently reading incorrect bias columns. This change carries the per-batch absolute cache origin into the CPU attention paths so resident column zero maps to the correct absolute bias column. Explicit `position_ids` remain consumed by RoPE before the cache-relative transition. ## Checklist - [x] Tests added/updated - [x] No breaking changes - [ ] Documentation updated (not applicable; no public API change)
Tianlei Wu (tianleiwu)
previously approved these changes
Sep 5, 2026
…32244) ### Description Adds support for using a plugin EP device's own allocator for input/output buffers in `onnxruntime_perf_test`, instead of always using the CPU allocator. - When a session uses plugin EP(s) (`--plugin_eps`), `onnxruntime_perf_test` now selects an allocator from the appended `OrtEpDevice`: it prefers the device's default (device-only) allocator, falls back to its host-accessible allocator, and falls back to the CPU allocator if neither is available or if more than one EP device was appended (no single unambiguous allocator to pick). - Generated inputs (`-r`) and pre-allocated fixed-shape outputs now use this allocator instead of the CPU allocator. - Real test data loaded from files is staged into the selected allocator's memory once at load time (via `Ort::Env::CopyTensor`), instead of being copied per-`Run()` implicitly. - String tensors are always kept in CPU-accessible memory (inputs and outputs), since `std::string` is placement-constructed by ORT directly into the tensor's owned memory and can't safely live in device-only memory. - Added `[Plugin EP]`-prefixed logging indicating which allocator was selected. ### Motivation and Context `onnxruntime_perf_test` previously always allocated input/output buffers on the CPU when running with plugin EPs. For plugin EPs whose kernels expect device-resident inputs/outputs, this meant every `Run()` iteration paid for an implicit host↔device copy that was entirely avoidable, skewing per-iteration latency measurements away from the EP's actual compute cost. Selecting and reusing the EP device's own allocator up front lets perf numbers better reflect steady-state inference cost for plugin EPs. --------- Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Backport the CoreML/libuuid build fix from #32268 so Linux Vcpkg builds do not depend on uuid-dev being preinstalled on the runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…32139) **Situation** `com.microsoft.GroupQueryAttention` requires the Value KV-cache in BNSH layout - `(batch_size, num_heads, sequence_length, head_size)` - for both the `past_value` input and the `present_value` output. Applications allocate those buffers themselves and bind them across decode steps. **Obstacle** Some execution providers execute GQA faster when the Value cache is BNHS - `(batch_size, num_heads, head_size, sequence_length)` - because the second attention matmul (`attn_weights @ V`) becomes an NT gemm. The operator schema cannot simply change: it is a stable contrib op and most EPs are BNSH-only. There was also no way for an application to discover an EP's preference, nor to tell a session which layout its buffers use. **Resolution** The GQA node stays BNSH and the conversion moves into the graph, where an EP compiler can absorb it: ``` past_value (BNHS, graph input) -> Transpose[0,1,3,2] -> GQA -> Transpose[0,1,3,2] -> present_value (BNHS, graph output) ``` Three pieces: 1. **EP advertises its preference.** New well-known `OrtEpDevice` metadata key `gqa_preferred_value_layout` (`kOrtEpDevice_EpMetadataKey_GqaPreferredValueLayout`), values `"BNSH"` (assumed when absent) or `"BNHS"`. No new C API - applications read it through the existing `OrtApi::EpDevice_EpMetadata`. 2. **Application selects the layout.** New session option `session.gqa_value_layout` (`kOrtSessionOptionsGqaValueLayout`), `"BNSH"` (default) or `"BNHS"`. Any other value fails session initialization. 3. **ORT core inserts the conversion.** New `GqaValueLayoutTransformer` inserts the two `Transpose(perm=[0,1,3,2])` nodes and swaps the last two dimensions of the `past_value` graph input and `present_value` graph output declared shapes, so `InferenceSession::ValidateInputsOutputs` accepts the application's buffers. An EP that reports `"BNHS"` fuses `Transpose -> GQA -> Transpose` into a single operation that reads BNHS directly and aliases `past_value`/`present_value` to one buffer, so the transposes never materialize. An EP that does not fuse them executes them: still correct, but a full copy of the Value cache in each direction per step and no past/present buffer sharing. A post-partitioning check logs a warning naming any GQA node whose flanking transposes survived, so that cost is diagnosable rather than silent. - The transformer is invoked directly from `InferenceSession::TransformGraph` rather than registered as a Level 1 optimizer. It must run at every optimization level including `ORT_DISABLE_ALL` (registered transformers at Level 1 and above are skipped there), and it must run *after* the Level 1 `TransposeOptimizer`, whose job is moving, merging and cancelling Transpose nodes, so the pattern reaches `GetCapability` intact. - Applied to the main graph only. Subgraphs (a BeamSearch decoder body, a Loop carried value) are not the application's boundary. Nodes whose `past_value` is not a graph input, or whose `present_value` is not a graph output, are skipped with a warning. - Idempotent. A model saved via `session.optimized_model_filepath` already carries the transform and may be reloaded with the option still set; the transformer detects the existing pattern and no-ops. - A 4-bit quantized Value cache is rejected with an error. Two 4-bit values are packed per byte along `head_size`, so a byte-wise `Transpose` cannot express the layout change and the declared-shape swap would be wrong. - `k_scale`/`v_scale` need no change. The GQA node is BNSH on both sides after the transform, so a `PER_CHANNEL` scale still has to broadcast against a BNSH tensor. Applications supply `v_scale` in the model-declared `[1, num_heads_k, 1, head_size]` shape regardless of the cache layout chosen. - The Key cache is unaffected. | File | Change | | --- | --- | | `include/onnxruntime/core/session/onnxruntime_ep_device_ep_metadata_keys.h` | New metadata key | | `include/onnxruntime/core/session/onnxruntime_session_options_config_keys.h` | New session option | | `onnxruntime/core/optimizer/gqa_value_layout_transformer.{h,cc}` | New transformer and the unfused-transpose diagnostic | | `onnxruntime/core/session/inference_session.cc` | Option validation, transformer invocation, post-partition diagnostic | | `onnxruntime/test/optimizer/gqa_value_layout_transformer_test.cc` | New tests | | `onnxruntime/test/autoep/library/example_plugin_ep/ep_factory.cc`, `onnxruntime/test/autoep/test_registration.cc` | Example EP advertises the key, with a round-trip assertion | | `docs/design/GQA_Value_Tensor_Layout.md` | Design document | Ten new cases are added, covering transpose insertion and boundary shape swapping, idempotency, the past-only and present-only variants, both skip conditions, the 4-bit rejection, and three session-level tests for the option plumbing — including that the transform applies at `ORT_DISABLE_ALL`, which pins down the placement decision. Behavior is unchanged unless the new session option is set to `"BNHS"`. Applications that manage a KV cache across decode steps (onnxruntime-genai and similar) must allocate the Value cache in whichever layout their target EP executes best, but ORT offered no mechanism to negotiate that. Without one, an EP whose GQA implementation prefers BNHS either gives up the gain or the application guesses, with no way to stay correct when a layer falls back to a BNSH-only provider. This change adds the negotiation - EP advertises, application selects, ORT core adapts the graph - while keeping the GQA schema and every existing kernel untouched. Expressing the layout change as ordinary `Transpose` nodes means correctness does not depend on the EP fusing them: a provider that cannot simply runs them. Follow-ups not included here, tracked in the design document: - CPU-fallback numerical parity tests. - The compiling EP's fusion support. - The ORT-format load path. `PartitionOrtFormatModel` does not go through `TransformGraph`, so for now the transform must be applied at conversion time, which the idempotency guard makes safe. --------- Co-authored-by: Tianlei Wu <tlwu@microsoft.com>
## Description Replace the obsolete `doxygen.nl` download URL for the pinned Doxygen 1.9.8 Linux binary with the official asset from the Doxygen GitHub release tag `Release_1_9_8`. The C API docs workflow has persistently failed while downloading Doxygen (runs [31917039689](https://github.com/microsoft/onnxruntime/actions/runs/31917039689), [31975339915](https://github.com/microsoft/onnxruntime/actions/runs/31975339915), and [32050624667](https://github.com/microsoft/onnxruntime/actions/runs/32050624667)). Because no fresh `onnxruntime-c-apidocs` artifact was produced before the previous artifact expired, the downstream ONNX Publish site run [32533178465](https://github.com/microsoft/onnxruntime/actions/runs/32533178465) failed at `Download C apidocs artifact`, blocking GitHub Pages deployment. This is intentionally separate from accessibility PR #32180. ## Validation - Confirmed the official release URL follows one redirect and returns HTTP 200. - Confirmed GitHub release tag `Release_1_9_8` contains `doxygen-1.9.8.linux.bin.tar.gz` (50,500,806 bytes). - Downloaded the archive successfully and verified gzip/tar integrity. - Confirmed extraction produces `doxygen-1.9.8/bin/doxygen` while preserving the workflow's existing extraction and invocation paths. - Parsed the workflow as valid YAML. - Dispatched the [C/C++ API docs workflow from this branch](https://github.com/microsoft/onnxruntime/actions/runs/32534100983); the install, Doxygen generation, and site staging steps succeeded, and the log reports `Doxygen version used: 1.9.8`. Artifact upload was skipped as expected because the ref is not `main`. ## Recovery after merge 1. Dispatch the **Update C/C++ API Docs** workflow on `main`. 2. Verify it uploads a fresh `onnxruntime-c-apidocs` artifact. 3. Rerun the failed GitHub Pages jobs from ONNX Publish site run 32533178465. No Doxygen version or unrelated workflow behavior is changed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Remove a newer-main TurboQuant header that was accidentally retained while resolving the #32139 test conflict. The header is not part of the 1.29 branch and is not used by the backported tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tianlei Wu (tianleiwu)
approved these changes
Sep 9, 2026
Tianlei Wu (tianleiwu)
enabled auto-merge (squash)
September 10, 2026 01:29
Hariharan Seshadri (hariharans29)
approved these changes
Sep 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This cherry-picks the following commits for the release: