Repository navigation
Attention masks are missing in SD3 to mask out text padding tokens #8673
Description
Describe the bug
In the attention implementation of SD3, attention masks currently are not used. This will result in inconsistent outputs for the different values max_seq_length where padding exists in text tokens as the attention scores of padding tokens are non-zero. This issue has been discussed in #8628, and is created to track the progress of fixing this problem.
Thanks @sayakpaul for the discussion.
Reproduction
n/a
Logs
No response
System Info
n/a
Who can help?
No response
Activity
rootonchair commentedon Jun 25, 2024
Hi @sayakpaul, I am interested in working on this issue
sayakpaul commentedon Jun 25, 2024
Thanks for your interest! Sure, let’s go.
github-actions commentedon Sep 14, 2024
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
SakshamDhawan commentedon Oct 29, 2024
Hi guys, if no one working on this, I'm willing to pick this up 👍
@sayakpaul
a-r-r-o-w commentedon Nov 20, 2024
Gentle ping to keep the activity going. @rootonchair Would you be able to contribute the fix?
rootonchair commentedon Nov 20, 2024
Yes @a-r-r-o-w , I will open a PR soon
github-actions commentedon Dec 15, 2024
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Bissmella commentedon Jan 8, 2026
Hi @sayakpaul,
I've investigated this issue. The attention mask is intentionally ignored in the text encoders (CLIP/T5), likely because the diffusion models were trained without masking padding tokens, possibly to preserve information about prompt length/detail (I would like to know more).
However, the core issue is in the latent transformers during diffusion. Different models concatenate
encoder_hidden_states(text) andhidden_states(image/noise latents) in different orders. As an example:Flux: [
encoder_hidden_states,hidden_states] see FluxSD3: [
hidden_states,encoder_hidden_states] see SD3This ordering difference explains why the PR mentioned above works for Flux but fails for SD3, as the attention mask positioning becomes model-dependent.
Proposed solution: Handle attention mask at the processor level (where q, k, v are constructed). Since each model has its own processor, we can implement model-specific mask preparation that respects the correct concatenation order and simply pass it to attention backbone.
sayakpaul commentedon Jan 9, 2026
I guess there's no way to know more unless someone from the Stability AI team can chime in here. Our integration followed the original codebase (like as always).
Your approach sounds okay to me but I guess if the model wasn't trained to handle masks, then I don't know how it's gonna play out. Maybe you can give it a try and compare outputs?