Definition
Attention is one of the key mechanisms that enables language models to use context when interpreting tokens.
Consider the word Apple:
- “I ate a tasty apple.” refers to the fruit.
- “Apple’s revenue increased.” refers to the company.
- “She is the apple of my eye.” has a different contextual meaning.
The word itself has not changed. The surrounding words provide the context needed to interpret it.
Attention allows a model to consider relationships between tokens and use relevant surrounding information when processing an input. This ability to capture context was a major breakthrough in modern language-model architectures.