Are the text padding tokens masked in SD3?

[va1bhavagrawal]

SD3 has the text embeddings coming from CLIP and T5 text encoders. These are padded upto the sequence length (77 for CLIP, max_sequence_length for T5).

How are the padding tokens masked? Do the image tokens directly attend to the padding tokens? I do not see any attention_mask being passed to the transformer? My intuition says that the image tokens should not be allowed to attend to the padding tokens.

Any ideas how this is handled?

[va1bhavagrawal]

I tried to visualize some image->text attention maps in SD 3.5 medium on random padding tokens. It seems that the attention maps are non zero, I have visualized some of them here:

Any explanation for why mask is not used for padding tokens?