Attention Is All You Need, 2017

In June 2017, eight authors from Google Brain and Google Research, led by Ashish Vaswani, posted Attention Is All You Need on arXiv. The paper proposed the Transformer, an architecture with no recurrence and no convolutions. Just attention, feed-forward layers, and residual connections.

The immediate target was machine translation, where Transformers beat the best recurrent models on English-to-German while training several times faster. Within two years, the paper would be one of the most cited in AI history.

Why Transformers took over

1. 1Parallel training: unlike RNNs, attention processes all positions at once, making full use of GPUs
2. 2Long-range dependencies: attention sees the whole sequence, not just a compressed state
3. 3Scaling: performance improved smoothly with more parameters, more data, and more compute
4. 4Generality: the same architecture worked for text, images, audio, code, and proteins

The Transformer became the common substrate for everything that followed. BERT from Google in 2018 used the encoder. GPT from OpenAI used the decoder. Vision Transformers, Whisper, AlphaFold, Stable Diffusion, and the current generation of chat models all rest on it.

“We propose a new simple network architecture, the Transformer, based solely on attention mechanisms.”

The big idea: one clean architectural choice, attention without recurrence, unlocked a decade of scaling. Every LLM you use today traces directly to this paper.