In the attention mechanism, what does the formula $$ ext{Attention}(Q,K,V) = ext{softmax}\left(\frac{QK^T}{\sqrt{d_k}} ight)V$$ represent?

Question

In the attention mechanism, what does the formula $$	ext{Attention}(Q,K,V) = 	ext{softmax}\left(\frac{QK^T}{\sqrt{d_k}}ight)V$$ represent?

MLQuiz · Accepted Answer

Introduced in *Attention Is All You Need* (Vaswani et al., 2017). The $$\sqrt{d_k}$$ scaling prevents dot products from growing too large, stabilising softmax gradients. [Read the paper](https://arxiv.org/abs/1706.03762).