Skip to content

Stabilize fork-sensitive nightly tests and AutoSP coverage - #8172

Merged
tohtana merged 6 commits into
deepspeedai:masterfrom
tohtana:tohtana/fix-nightly-zero-nvme-hang
Jul 29, 2026
Merged

Stabilize fork-sensitive nightly tests and AutoSP coverage#8172
tohtana merged 6 commits into
deepspeedai:masterfrom
tohtana:tohtana/fix-nightly-zero-nvme-hang

Conversation

@tohtana

@tohtana tohtana commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator

This PR fixes issues with the nightly full test:

  • Fork-sensitive tests: Run these tests in the non-forked sequential phase
    • Heavy two-rank NVMe cases contended in the forked parallel phase
    • The CPU-only FlopsProfiler formatting test could inherit initialized CUDA state and fail in a forked subprocess
  • AutoSP coverage: TestSDPANodesCompile inspected only the first FX graph produced from a Transformers model, but a compiled model can be split into multiple graphs and the first graph may not contain attention. The test now constructs an FX GraphModule with Q/K/V inputs and one torch.nn.functional.scaled_dot_product_attention call. The separate four-GPU tests still continue to use a tiny Transformers Llama model.
  • Transformers version output: Nightly tests install Transformers from Git, but the workflow incorrectly displayed the unused PyPI fallback version (4.50.0) as if it were installed. It now reports the actually installed version and Git commit, shows unused fields as N/A.

tohtana added 4 commits July 19, 2026 10:36
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
Signed-off-by: Masahiro Tanaka <mtanaka@anyscale.com>
@tohtana
tohtana requested review from loadams and tjruwase as code owners July 24, 2026 05:44
tohtana and others added 2 commits July 28, 2026 11:28

@delock delock left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@tohtana
tohtana added this pull request to the merge queue Jul 29, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jul 29, 2026
@tohtana
tohtana added this pull request to the merge queue Jul 29, 2026
Merged via the queue into deepspeedai:master with commit 86fd223 Jul 29, 2026
13 checks passed
@tohtana
tohtana deleted the tohtana/fix-nightly-zero-nvme-hang branch July 29, 2026 06:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants