Skip to content

Support flash-attn 2.7.0 in FPDT attention - #8022

Merged
tohtana merged 2 commits into
deepspeedai:masterfrom
xbcReal:fix/flash-attn-2.7-fpdt-compat
May 24, 2026
Merged

Support flash-attn 2.7.0 in FPDT attention#8022
tohtana merged 2 commits into
deepspeedai:masterfrom
xbcReal:fix/flash-attn-2.7-fpdt-compat

Conversation

@xbcReal

@xbcReal xbcReal commented May 22, 2026

Copy link
Copy Markdown
Contributor

Summary

  • flash-attn 2.7.0 split the window_size argument into window_size_left/window_size_right ints, and reduced _flash_attn_forward's return from 8 values to 4. Calling FPDT with flash-attn >= 2.7 currently errors out.
  • Add a >= 2.7.0 branch to both _FPDTGPUAttentionImpl_ and _FPDTGPUOffloadingAttentionImpl_ (forward + backward) so FPDT keeps working with newer flash-attn releases while remaining compatible with 2.6.x and earlier.
  • Update FPDT_Attention's ImportError message so it no longer pins 2.6.3.

flash-attn 2.7.0 split the window_size argument into separate
window_size_left/window_size_right ints, and reduced the
_flash_attn_forward return from 8 values to 4. Add a 2.7.0+ branch
to both _FPDTGPUAttentionImpl_ and _FPDTGPUOffloadingAttentionImpl_
(forward + backward) so FPDT keeps working with newer flash-attn
releases while remaining compatible with 2.6.x and earlier.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: binchengxiong <binchengxiong@alibaba-inc.com>
@xbcReal
xbcReal requested a review from tohtana as a code owner May 22, 2026 03:13
@xbcReal

xbcReal commented May 22, 2026

Copy link
Copy Markdown
Contributor Author

If we don't add the uploaded commit, we would come across the following error message when running the test command using flash-attn 2.7.0 version:
pytest -v sequence_parallelism/test_ulysses.py::TestFPDTAttention::test_FPDT_attention_offloading_output_consistency[32-8-128-2048-4]

RuntimeError: flash_attn::_flash_attn_forward() is missing value for argument 'window_size_left'. Declaration: flash_attn::_flash_attn_forward(Tensor q, Tensor k, Tensor v, float dropout_p, float softmax_scale, bool causal, SymInt window_size_left, SymInt window_size_right, float softcap, Tensor? alibi_slopes, bool return_softmax) -> (Tensor, Tensor, Tensor, Tensor)

@tohtana

tohtana commented May 22, 2026

Copy link
Copy Markdown
Collaborator

@YJHMITWEB Can you review this PR?

@YJHMITWEB

Copy link
Copy Markdown
Collaborator

@YJHMITWEB Can you review this PR?

Hi @tohtana , this PR looks good to me, as it only includes a version check. We can bring it in with a rebase.

@tohtana tohtana left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @xbcReal! Let me merge this.

@YJHMITWEB Thank you for your review!

@tohtana
tohtana merged commit 4542922 into deepspeedai:master May 24, 2026
9 checks passed
frozenleaves pushed a commit to frozenleaves/DeepSpeed that referenced this pull request Jun 8, 2026
## Summary
- flash-attn 2.7.0 split the `window_size` argument into
`window_size_left`/`window_size_right` ints, and reduced
`_flash_attn_forward`'s return from 8 values to 4. Calling FPDT with
flash-attn >= 2.7 currently errors out.
- Add a `>= 2.7.0` branch to both `_FPDTGPUAttentionImpl_` and
`_FPDTGPUOffloadingAttentionImpl_` (forward + backward) so FPDT keeps
working with newer flash-attn releases while remaining compatible with
2.6.x and earlier.
- Update `FPDT_Attention`'s ImportError message so it no longer pins
2.6.3.

Signed-off-by: binchengxiong <binchengxiong@alibaba-inc.com>
Co-authored-by: binchengxiong <binchengxiong@alibaba-inc.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
nathon-lee pushed a commit to nathon-lee/DeepSpeed_woo that referenced this pull request Jul 1, 2026
## Summary
- flash-attn 2.7.0 split the `window_size` argument into
`window_size_left`/`window_size_right` ints, and reduced
`_flash_attn_forward`'s return from 8 values to 4. Calling FPDT with
flash-attn >= 2.7 currently errors out.
- Add a `>= 2.7.0` branch to both `_FPDTGPUAttentionImpl_` and
`_FPDTGPUOffloadingAttentionImpl_` (forward + backward) so FPDT keeps
working with newer flash-attn releases while remaining compatible with
2.6.x and earlier.
- Update `FPDT_Attention`'s ImportError message so it no longer pins
2.6.3.

Signed-off-by: binchengxiong <binchengxiong@alibaba-inc.com>
Co-authored-by: binchengxiong <binchengxiong@alibaba-inc.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Signed-off-by: nathon-lee <leejianwoo@gmail.com>
nathon-lee pushed a commit to nathon-lee/DeepSpeed_woo that referenced this pull request Jul 1, 2026
## Summary
- flash-attn 2.7.0 split the `window_size` argument into
`window_size_left`/`window_size_right` ints, and reduced
`_flash_attn_forward`'s return from 8 values to 4. Calling FPDT with
flash-attn >= 2.7 currently errors out.
- Add a `>= 2.7.0` branch to both `_FPDTGPUAttentionImpl_` and
`_FPDTGPUOffloadingAttentionImpl_` (forward + backward) so FPDT keeps
working with newer flash-attn releases while remaining compatible with
2.6.x and earlier.
- Update `FPDT_Attention`'s ImportError message so it no longer pins
2.6.3.

Signed-off-by: binchengxiong <binchengxiong@alibaba-inc.com>
Co-authored-by: binchengxiong <binchengxiong@alibaba-inc.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: Masahiro Tanaka <81312776+tohtana@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants