Question about attention weight

[HopeZ123]

Dear authors,

Thank you so much for your great work. May I ask what is the real meaning of attention weights? Do they mean the feature maps of different attention layers or the query, key, value weights or parameters? I am so confused with this, I do not exactly know what I should visualize, should I save the feature maps from different attention layers and then draw the heat maps for these layers? I think it seems like this in the figure shown in your paper. Look forward to your reply. Thank you so much.

[LPXTT]

Hello, thanks for your interests. The visualization maps are calculated based on the attention maps. First, we save the target and search tokens from different layers. Then, we calculate the target-related attention map. Suppose that we have 16 target tokens and 64 search tokens, the attention map will be 8080. We select a 6416 attention map from it, 64 is the search tokens from the first 80 and 16 is the target tokens from the last 80 (if we put the target tokens in front of the search tokens). After that, we calculate the average values along the second dimension of the 6416 attention map. Then, we will get 641 maps which can be reshaped to 8*8. That is our heat map.

[HopeZ123]

Dear Author, Thank you so much for your explanation, have a nice day. Regards, Shuo

…

On Sun, Apr 16, 2023 at 6:10 AM LPXTT ***@***.***> wrote: Hello, thanks for your interests. The visualization maps are calculated based on the attention maps. First, we save the target and search tokens from different layers. Then, we calculate the target-related attention map. Suppose that we have 16 target tokens and 64 search tokens, the attention map will be 80*80. We select a 64*16 attention map from it, 64 is the search tokens from the first 80 and 16 is the target tokens from the last 80 (if we put the target tokens in front of the search tokens). After that, we calculate the average values along the second dimension of the 64*16 attention map. Then, we will get 64*1 maps which can be reshaped to 8*8. That is our heat map. — Reply to this email directly, view it on GitHub <#11 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A7AU7KQDVFZMLS76ZZIDAELXBPAP7ANCNFSM6AAAAAAWV7CFKM> . You are receiving this because you authored the thread.Message ID: ***@***.***>