Attention Is All You Need (Original Transformer Paper)
by Vaswani et al., Google (NeurIPS 2017)
Fifteen pages that define the architecture behind every modern LLM.
Overview
The 2017 NeurIPS paper by Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser and Illia Polosukhin that introduced the Transformer, the architecture every current large language model descends from. Its argument is that recurrence and convolution are unnecessary: a network built solely from attention can relate all positions in a sequence in a small, constant number of steps regardless of the distance between them, where a recurrent network needs a number of steps that grows with that distance and a convolutional one still grows, just more slowly. The paper defines scaled dot-product attention, multi-head attention, sinusoidal positional encodings, the encoder-decoder stack, position-wise feed-forward layers and the residual-plus-layer-normalisation pattern that every subsequent variant inherits, so reading it once makes the architecture diagrams in later papers legible. Results are reported on machine translation: 28.4 BLEU on WMT 2014 English-to-German and 41.8 BLEU on English-to-French, the latter a single-model state of the art trained in three and a half days on eight GPUs, which is the parallelism argument made concrete rather than asserted. Google released the Tensor2Tensor library alongside the paper so the results were reproducible from a few commands. It is fifteen pages, freely available on arXiv at revision v7 last updated August 2023, and the attention and positional-encoding sections repay a second pass with a code implementation open beside them.
At a Glance
- Topic
- Models
- Level
- Advanced
- Format
- Paper
- Cost
- Free
- Duration
- 15 pages, ~2-3 hours for a careful first read
- Provider
- Vaswani et al., Google (NeurIPS 2017)
- Hands-on
- No
- Certificate
- None
What You’ll Learn
- ✓How scaled dot-product attention computes weighted relationships between all sequence positions
- ✓Why multi-head attention lets one layer attend to several representation subspaces
- ✓How positional encodings inject order into an architecture with no recurrence
- ✓The encoder-decoder stack layout that later GPT and BERT variants simplify
- ✓Why constant path length in attention parallelises where recurrent models cannot
- ✓How masked self-attention prevents a decoder from attending to future tokens
- ✓Compare the complexity trade-offs between self-attention, recurrent and convolutional layers
Highlights
- •The source text for every modern LLM architecture, which later papers assume you have read
- •Fifteen pages and freely available: the highest ratio of influence to reading time in the field
- •Introduces scaled dot-product attention, multi-head attention and positional encoding in their original notation
- •Backs the parallelism claim with concrete numbers: 28.4 BLEU English-German, 41.8 English-French, 3.5 days on eight GPUs
- •Google shipped the Tensor2Tensor library alongside it, so the architecture was reproducible from day one
Who It’s For
Best For
- ✓Engineers who use transformers daily but have never read the original definition
- ✓ML practitioners preparing to read modern architecture papers without hand-waving
- ✓Anyone implementing attention from scratch as a learning exercise
Prerequisites
- •Linear algebra: matrix multiplication, dot products and softmax over vectors
- •Familiarity with neural network training, including residual connections and layer normalisation
- •Helpful but not required: prior exposure to sequence-to-sequence models and RNNs
FAQ
What is Attention Is All You Need (Original Transformer Paper)?
The 2017 NeurIPS paper that introduced the Transformer and dispensed with recurrence and convolution entirely. It defines scaled dot-product attention, multi-head attention and positional encoding, the vocabulary every architecture paper since assumes you already know. Fifteen pages, free on arXiv, and the fastest way to stop treating the attention block in your stack as a black box you inherited.
Is Attention Is All You Need (Original Transformer Paper) free?
Attention Is All You Need (Original Transformer Paper) is free to access.
What level is Attention Is All You Need (Original Transformer Paper) for?
Attention Is All You Need (Original Transformer Paper) is aimed at a advanced audience. Recommended background: Linear algebra: matrix multiplication, dot products and softmax over vectors, Familiarity with neural network training, including residual connections and layer normalisation, Helpful but not required: prior exposure to sequence-to-sequence models and RNNs.
How long does Attention Is All You Need (Original Transformer Paper) take?
Expect roughly 15 pages, ~2-3 hours for a careful first read. Most learners work through it at their own pace.
What will I learn from Attention Is All You Need (Original Transformer Paper)?
You'll learn: How scaled dot-product attention computes weighted relationships between all sequence positions; Why multi-head attention lets one layer attend to several representation subspaces; How positional encodings inject order into an architecture with no recurrence; The encoder-decoder stack layout that later GPT and BERT variants simplify; Why constant path length in attention parallelises where recurrent models cannot; How masked self-attention prevents a decoder from attending to future tokens; Compare the complexity trade-offs between self-attention, recurrent and convolutional layers.
Topics
Sources
This page was written from 3 sources, 2 on domains other than arxiv.org.