Attention (machine learning)

{{See also|Timeline of machine learning}}

Academic reviews of the history of the attention mechanism are provided in Niu et al.{{Cite journal |last1=Niu |first1=Zhaoyang |last2=Zhong |first2=Guoqiang |last3=Yu |first3=Hui |date=2021-09-10 |title=A review on the attention mechanism of deep learning |url=https://www.sciencedirect.com/science/article/pii/S092523122100477X |journal=Neurocomputing |volume=452 |pages=48–62 |doi=10.1016/j.neucom.2021.03.091 |issn=0925-2312|url-access=subscription }} and Soydaner.{{Cite journal |last=Soydaner |first=Derya |date=August 2022 |title=Attention mechanism in neural networks: where it comes and where it goes |url=https://link.springer.com/10.1007/s00521-022-07366-3 |journal=Neural Computing and Applications |language=en |volume=34 |issue=16 |pages=13371–13385 |arxiv=2204.13154 |doi=10.1007/s00521-022-07366-3 |issn=0941-0643}}

The modern era of machine attention was revitalized by grafting an attention mechanism (Fig 1. orange) to an Encoder-Decoder.

|}

Figure 2 shows the internal step-by-step operation of the attention block (A) in Fig 1.

{{Image frame|content=700px|width=700|align=center|caption=Figure 2. The diagram shows the attention forward pass calculating correlations of the word "that" with other words in "See that girl run." Given the right weights from training, the network should be able to identify "girl" as a highly correlated word. Some things to note:

• This example focuses on the attention of a single word "that". In practice, the attention of each word is calculated in parallel to speed up calculations. Simply changing the lowercase "x" vector to the uppercase "X" matrix will yield the formula for this.
• Softmax scaling {{math|qW_k^T}} / {{radic|100}} prevents a high variance in {{math|qW_k^T}} that would allow a single word to excessively dominate the softmax resulting in attention to only one word, as a discrete hard max would do.
• Notation: the commonly written row-wise {{math|softmax}} formula above assumes that vectors are rows, which runs contrary to the standard math notation of column vectors. More correctly, we should take the transpose of the context vector and use the column-wise {{math|softmax}}, resulting in the more correct form

$$\backslash begin\{align\}\; (XW\_v)^T\; *\; \{[\; (W\_k\; X^T)\; *\; \{\; (\backslash underline\{x\}W\_q)^T\; \}\; ]\_\{sm\}\; \}\; \backslash end\{align\}$$.}}

This attention scheme has been compared to the Query-Key analogy of relational databases. That comparison suggests an asymmetric role for the Query and Key vectors, where one item of interest (the Query vector "that") is matched against all possible items (the Key vectors of each word in the sentence). However, both Self and Cross Attentions' parallel calculations matches all tokens of the K matrix with all tokens of the Q matrix; therefore the roles of these vectors are symmetric. Possibly because the simplistic database analogy is flawed, much effort has gone into understanding attention mechanisms further by studying their roles in focused settings, such as in-context learning, masked language tasks, stripped down transformers, bigram statistics, N-gram statistics, pairwise convolutions, and arithmetic factoring.

In translating between languages, alignment is the process of matching words from the source sentence to words of the translated sentence. Networks that perform verbatim translation without regard to word order would show the highest scores along the (dominant) diagonal of the matrix. The off-diagonal dominance shows that the attention mechanism is more nuanced.

Consider an example of translating I love you to French. On the first pass through the decoder, 94% of the attention weight is on the first English word I, so the network offers the word je. On the second pass of the decoder, 88% of the attention weight is on the third English word you, so it offers t'. On the last pass, 95% of the attention weight is on the second English word love, so it offers aime.

In the I love you example, the second word love is aligned with the third word aime. Stacking soft row vectors together for je, t', and aime yields an alignment matrix:

Sometimes, alignment can be multiple-to-multiple. For example, the English phrase look it up corresponds to cherchez-le. Thus, "soft" attention weights work better than "hard" attention weights (setting one attention weight to 1, and the others to 0), as we would like the model to make a context vector consisting of a weighted sum of the hidden vectors, rather than "the best one", as there may not be a best hidden vector.

File:Self-attention_in_CNN,_RNN,_and_self-attention.svg

Many variants of attention implement soft weights, such as

• fast weight programmers, or fast weight controllers (1992). A "slow" neural network outputs the "fast" weights of another neural network through outer products. The slow network learns by gradient descent. It was later renamed as "linearized self-attention".
• Bahdanau-style attention, also referred to as additive attention,
• Luong-style attention, which is known as multiplicative attention,
• highly parallelizable self-attention introduced in 2016 as decomposable attention{{cite arXiv |eprint=1601.06733 |class=cs.CL |first1=Jianpeng |last1=Cheng |first2=Li |last2=Dong |title=Long Short-Term Memory-Networks for Machine Reading |date=2016-09-20 |last3=Lapata |first3=Mirella}} and successfully used in transformers a year later,
• positional attention and factorized positional attention.

For convolutional neural networks, attention mechanisms can be distinguished by the dimension on which they operate, namely: spatial attention, channel attention, or combinations.

These variants recombine the encoder-side inputs to redistribute those effects to each target output. Often, a correlation-style matrix of dot products provides the re-weighting coefficients. In the figures below, W is the matrix of context attention weights, similar to the formula in Core Calculations section above.

The size of the attention matrix is proportional to the square of the number of input tokens. Therefore, when the input is long, calculating the attention matrix requires a lot of GPU memory. Flash attention is an implementation that reduces the memory needs and increases efficiency without sacrificing accuracy. It achieves this by partitioning the attention computation into smaller blocks that fit into the GPU's faster on-chip memory, reducing the need to store large intermediate matrices and thus lowering memory usage while increasing computational efficiency.{{Cite web |last=Mittal |first=Aayush |date=2024-07-17 |title=Flash Attention: Revolutionizing Transformer Efficiency |url=https://www.unite.ai/flash-attention-revolutionizing-transformer-efficiency/ |access-date=2024-11-16 |website=Unite.AI |language=en-US}}

Flex Attention https://pytorch.org/blog/flexattention/ is an attention kernel developed by Meta that allows users to modify attention scores prior to softmax and dynamically chooses the optimal attention algorithm.

The major breakthrough came with self-attention, where each element in the input sequence attends to all others, enabling the model to capture global dependencies. This idea was central to the Transformer architecture, which replaced recurrence entirely with attention mechanisms. As a result, Transformers became the foundation for models like BERT, GPT, and T5 (Vaswani et al., 2017).

Attention is widely used in natural language processing, computer vision, and speech recognition. In NLP, it improves context understanding in tasks like question answering and summarization. In vision, visual attention helps models focus on relevant image regions, enhancing object detection and image captioning.

For matrices: $\backslash mathbf\{Q\}\backslash in\backslash mathbb\{R\}^\{m\backslash times\; d\_k\},\; \backslash mathbf\{K\}\backslash in\backslash mathbb\{R\}^\{n\backslash times\; d\_k\}$ and $\backslash mathbf\{V\}\backslash in\backslash mathbb\{R\}^\{n\backslash times\; d\_v\}$, the scaled dot-product, or QKV attention is defined as:

$$\backslash text\{Attention\}(\backslash mathbf\{Q\},\; \backslash mathbf\{K\},\; \backslash mathbf\{V\})\; =\; \backslash text\{softmax\}\backslash left(\backslash frac\{\backslash mathbf\{Q\}\backslash mathbf\{K\}^T\}\{\backslash sqrt\{d\_k\}\}\backslash right)\backslash mathbf\{V\}\backslash in\backslash mathbb\{R\}^\{m\backslash times\; d\_v\}$$

where $\{\}^T$ denotes transpose and the softmax function is applied independently to every row of its argument. The matrix $\backslash mathbf\{Q\}$ contains $m$ queries, while matrices $\backslash mathbf\{K\},\; \backslash mathbf\{V\}$ jointly contain an unordered set of $n$ key-value pairs. Value vectors in matrix $\backslash mathbf\{V\}$ are weighted using the weights resulting from the softmax operation, so that the rows of the $m$-by-$d\_v$ output matrix are confined to the convex hull of the points in $\backslash mathbb\{R\}^\{d\_v\}$ given by the rows of $\backslash mathbf\{V\}$.

To understand the permutation invariance and permutation equivariance properties of QKV attention,{{cite arXiv |last1=Lee |first1=Juho |last2=Lee |first2=Yoonho |last3=Kim |first3=Jungtaek |last4=Kosiorek |first4=Adam R |last5=Choi |first5=Seungjin |last6=Teh |first6=Yee Whye |title=Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks |date=2018 |class=cs.LG |eprint=1810.00825 }} let $\backslash mathbf\{A\}\backslash in\backslash mathbb\{R\}^\{m\backslash times\; m\}$ and $\backslash mathbf\{B\}\backslash in\backslash mathbb\{R\}^\{n\backslash times\; n\}$ be permutation matrices; and $\backslash mathbf\{D\}\backslash in\backslash mathbb\{R\}^\{m\backslash times\; n\}$ an arbitrary matrix. The softmax function is permutation equivariant in the sense that:

:$$

\text{softmax}(\mathbf{A}\mathbf{D}\mathbf{B}) = \mathbf{A}\,\text{softmax}(\mathbf{D})\mathbf{B}

By noting that the transpose of a permutation matrix is also its inverse, it follows that:

:$$

\text{Attention}(\mathbf{A}\mathbf{Q}, \mathbf{B}\mathbf{K}, \mathbf{B}\mathbf{V}) = \mathbf{A}\,\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V})

which shows that QKV attention is equivariant with respect to re-ordering the queries (rows of $\backslash mathbf\{Q\}$); and invariant to re-ordering of the key-value pairs in $\backslash mathbf\{K\},\backslash mathbf\{V\}$. These properties are inherited when applying linear transforms to the inputs and outputs of QKV attention blocks. For example, a simple self-attention function defined as:

:$$

\mathbf{X}\mapsto\text{Attention}(\mathbf{X}\mathbf{T}_q, \mathbf{X}\mathbf{T}_k, \mathbf{X}\mathbf{T}_v)

is permutation equivariant with respect to re-ordering the rows of the input matrix $X$ in a non-trivial way, because every row of the output is a function of all the rows of the input. Similar properties hold for multi-head attention, which is defined below.

When QKV attention is used as a building block for an autoregressive decoder, and when at training time all input and output matrices have $n$ rows, a masked attention variant is used:

\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}}+\mathbf{M}\right)\mathbf{V}

where the mask, $\backslash mathbf\{M\}\backslash in\backslash mathbb\{R\}^\{n\backslash times\; n\}$ is a strictly upper triangular matrix, with zeros on and below the diagonal and $-\backslash infty$ in every element above the diagonal. The softmax output, also in $\backslash mathbb\{R\}^\{n\backslash times\; n\}$ is then lower triangular, with zeros in all elements above the diagonal. The masking ensures that for all

File:Encoder cross-attention, multiheaded version.png

Multi-head attention

\text{MultiHead}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{Concat}(\text{head}_1, ..., \text{head}_h)\mathbf{W}^O

where each head is computed with QKV attention as:

\text{head}_i = \text{Attention}(\mathbf{Q}\mathbf{W}_i^Q, \mathbf{K}\mathbf{W}_i^K, \mathbf{V}\mathbf{W}_i^V)

and $\backslash mathbf\{W\}\_i^Q,\; \backslash mathbf\{W\}\_i^K,\; \backslash mathbf\{W\}\_i^V$, and $\backslash mathbf\{W\}^O$ are parameter matrices.

The permutation properties of (standard, unmasked) QKV attention apply here also. For permutation matrices, $\backslash mathbf\{A\},\; \backslash mathbf\{B\}$:

:$$

\text{MultiHead}(\mathbf{A}\mathbf{Q}, \mathbf{B}\mathbf{K}, \mathbf{B}\mathbf{V}) = \mathbf{A}\,\text{MultiHead}(\mathbf{Q}, \mathbf{K}, \mathbf{V})

from which we also see that multi-head self-attention:

:$$

\mathbf{X}\mapsto\text{MultiHead}(\mathbf{X}\mathbf{T}_q, \mathbf{X}\mathbf{T}_k, \mathbf{X}\mathbf{T}_v)

is equivariant with respect to re-ordering of the rows of input matrix $X$.

\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}(\tanh(\mathbf{W}_Q\mathbf{Q} + \mathbf{W}_K\mathbf{K})\mathbf{V})

where $\backslash mathbf\{W\}\_Q$ and $\backslash mathbf\{W\}\_K$ are learnable weight matrices.

\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}(\mathbf{Q}\mathbf{W}\mathbf{K}^T)\mathbf{V}

where $\backslash mathbf\{W\}$ is a learnable weight matrix.

Self-attention is essentially the same as cross-attention, except that query, key, and value vectors all come from the same model. Both encoder and decoder can use self-attention, but with subtle differences.

For encoder self-attention, we can start with a simple encoder without self-attention, such as an "embedding layer", which simply converts each input word into a vector by a fixed lookup table. This gives a sequence of hidden vectors $h\_0,\; h\_1,\; \backslash dots$. These can then be applied to a dot-product attention mechanism, to obtain$$\backslash begin\{aligned\}$$

h_0' &= \mathrm{Attention}(h_0 W^Q, HW^K, H W^V) \\

h_1' &= \mathrm{Attention}(h_1 W^Q, HW^K, H W^V) \\

&\cdots

\end{aligned} or more succinctly, $H\text{'}\; =\; \backslash mathrm\{Attention\}(H\; W^Q,\; HW^K,\; H\; W^V)$. This can be applied repeatedly, to obtain a multilayered encoder. This is the "encoder self-attention", sometimes called the "all-to-all attention", as the vector at every position can attend to every other.

File:Decoder self-attention with causal masking, detailed diagram.pngFor decoder self-attention, all-to-all attention is inappropriate, because during the autoregressive decoding process, the decoder cannot attend to future outputs that has yet to be decoded. This can be solved by forcing the attention weights $w\_\{ij\}\; =\; 0$ for all , called "causal masking". This attention mechanism is the "causally masked self-attention".

• Recurrent neural network
• seq2seq
• Transformer (deep learning architecture)
• Attention
• Dynamic neural network

{{Reflist|refs=

{{ cite web | url = https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html | title = Pytorch.org seq2seq tutorial | access-date=December 2, 2021 }}

{{Cite book| publisher = MIT Press | isbn = 978-0-262-68053-0 |editor1-last=Rumelhart | editor1-first = David E. |editor2-last=Hinton |editor2-first=G. E. | editor3=PDP Research Group |last1=Rumelhart | first1 = David E. |last2=Hinton |first2=G. E. | last3 = Mcclelland| first3 = James L. |chapter=A General Framework for Parallel Distributed Processing | title = Parallel Distributed Processing, Volume 1: Explorations in the Microstructure of Cognition: Foundations | location = Cambridge, Massachusetts | date = 1987-07-29|chapter-url = https://stanford.edu/~jlmcc/papers/PDP/Chapter2.pdf| language = en}}

{{Cite journal |last1=Schmidhuber|first1=Jürgen|author-link1=Jürgen Schmidhuber|date=1992|title=Learning to control fast-weight memories: an alternative to recurrent nets.|journal=Neural Computation|volume=4|issue=1 |pages=131–139|doi=10.1162/neco.1992.4.1.131 |s2cid=16683347 }}

{{Cite conference | last1=Schlag|first1=Imanol| last2=Irie|first2=Kazuki| last3=Schmidhuber|first3=Jürgen|author-link1=Juergen Schmidhuber|title= Linear Transformers Are Secretly Fast Weight Programmers|publisher=Springer|date=2021 |pages=9355–9366 |book-title= ICML 2021}}

{{ cite arXiv |eprint=2407.12034 |last1=Nguyen |first1=Timothy |title=Understanding Transformers via N-gram Statistics |date=2024 |class=cs.CL }}

{{cite arXiv|last1=Britz|first1=Denny|last2=Goldie|first2=Anna|last3=Luong|first3=Minh-Thanh|last4=Le|first4=Quoc|date=2017-03-21|title=Massive Exploration of Neural Machine Translation Architectures|class=cs.CV|eprint=1703.03906}}

{{ cite arXiv|last1=Luong|first1=Minh-Thang| date=2015-09-20 |title=Effective Approaches to Attention-Based Neural Machine Translation |class=cs.CL| eprint=1508.04025v5}}

{{cite AV media |people = Neil Rhodes |date=2021 |title= CS 152 NN—27: Attention: Keys, Queries, & Values | url=https://www.youtube.com/watch?v=rA28vBqN4RM | access-date=2021-12-22 | time=06:30 }}

{{cite AV media |people = Alfredo Canziani & Yann Lecun |date=2021 |title= NYU Deep Learning course, Spring 2020 | url=https://www.youtube.com/watch?v=f01J0Dri-6k | access-date=2021-12-22 | time=05:30 }}

{{cite AV media |people = Alfredo Canziani & Yann Lecun |date=2021 |title= NYU Deep Learning course, Spring 2020 | url=https://www.youtube.com/watch?v=f01J0Dri-6k | access-date=2021-12-22 | time=20:15 }}

{{ cite web|url=https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html | title = NLP From Scratch: Translation With a Sequence To Sequence Network and Attention | last=Robertson|first=Sean| website=pytorch.org|access-date=2021-12-22 }}

{{cite arXiv |last1=Bahdanau |first1=Dzmitry |last2=Cho |first2=Kyunghyun |last3=Bengio |first3=Yoshua |title=Neural Machine Translation by Jointly Learning to Align and Translate |date=2014 |class=cs.CL |eprint=1409.0473}}

{{Cite arXiv |last1=Woo |first1=Sanghyun |last2=Park |first2=Jongchan |last3=Lee |first3=Joon-Young |last4=Kweon |first4=In So |date=2018-07-18 |title=CBAM: Convolutional Block Attention Module |class=cs.CV |eprint=1807.06521 }}

{{Cite arXiv|last1=Georgescu |first1=Mariana-Iuliana |last2=Ionescu |first2=Radu Tudor |last3=Miron |first3=Andreea-Iuliana |last4=Savencu |first4=Olivian |last5=Ristea |first5=Nicolae-Catalin |last6=Verga |first6=Nicolae |last7=Khan |first7=Fahad Shahbaz |date=2022-10-12 |title=Multimodal Multi-Head Convolutional Attention with Various Kernel Sizes for Medical Image Super-Resolution |class=eess.IV |eprint=2204.04218 }}

{{Cite book |last1=Hu |first1=Jie |last2=Shen |first2=Li |last3=Sun |first3=Gang |title=2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition |chapter=Squeeze-and-Excitation Networks |year=2018 |chapter-url=https://ieeexplore.ieee.org/document/8578843 |pages=7132–7141 |doi=10.1109/CVPR.2018.00745|arxiv=1709.01507 |isbn=978-1-5386-6420-9 |s2cid=206597034 }}

{{Cite book |last1=Zhu |first1=Xizhou |last2=Cheng |first2=Dazhi |last3=Zhang |first3=Zheng |last4=Lin |first4=Stephen |last5=Dai |first5=Jifeng |title=2019 IEEE/CVF International Conference on Computer Vision (ICCV) |chapter=An Empirical Study of Spatial Attention Mechanisms in Deep Networks |year=2019 |chapter-url=https://ieeexplore.ieee.org/document/9009578 |pages=6687–6696 |doi=10.1109/ICCV.2019.00679|arxiv=1904.05873 |isbn=978-1-7281-4803-8 |s2cid=118673006 }}

{{Cite web|url=https://www.catalyzex.com/paper/learning-positional-attention-for-sequential|title=Learning Positional Attention for Sequential Recommendation

|website=catalyzex.com}}

{{cite journal |last1=Zhang |first1=Ruiqi |title= Trained Transformers Learn Linear Models In-Context

|journal=Journal of Machine Learning Research 1-55 |date=2024 |volume=25 |arxiv=2306.09927 |url=https://jmlr.org/papers/volume25/23-1042/23-1042.pdf}}

{{cite journal|arxiv=2304.07235|last1=Rende |first1=Riccardo |title=Mapping of attention mechanisms to a generalized Potts model |journal=Physical Review Research |year=2024 |volume=6 |issue=2 |page=023057 |doi=10.1103/PhysRevResearch.6.023057 |bibcode=2024PhRvR...6b3057R }}

{{cite arXiv|eprint=2311.01906|last1=He |first1=Bobby |title=Simplifying Transformers Blocks |year=2023 |class=cs.LG }}

{{ cite web|url=https://transformer-circuits.pub | title = Transformer Circuits |website=transformer-circuits.pub }}

{{cite arXiv|eprint=2308.15594|last1=Charton |first1=François |title=Learning the Greatest Common Divisor: Explaining Transformer Predictions |year=2023 |class=cs.LG }}

{{cite AV media |date=2023 |title=Transformer Neural Network Derived From Scratch | url=https://www.youtube.com/watch?v=kWLed8o5M2Y&t=330s | access-date=2024-04-07 | time=05:30 }}

}}

• {{cite journal |last1=Olah |first1=Chris |last2=Carter |first2=Shan |date=September 8, 2016 |title=Attention and Augmented Recurrent Neural Networks |url=https://distill.pub/2016/augmented-rnns/ |journal=Distill |publisher=Distill Working Group |volume=1 |issue=9 |doi=10.23915/distill.00001}}
• Dan Jurafsky and James H. Martin (2022) [https://web.stanford.edu/~jurafsky/slp3/ Speech and Language Processing (3rd ed. draft, January 2022)], ch. 10.4 Attention and ch. 9.7 Self-Attention Networks: Transformers
• Alex Graves (4 May 2020), [https://www.youtube.com/watch?v=AIiwuClvH6k&vl=en-GB Attention and Memory in Deep Learning] (video lecture), DeepMind / UCL, via YouTube

{{Artificial intelligence navbox}}

Category:Machine learning