Attention Is All You Need

Onyx Okay so we're going back to the paper that basically everything since is built on, and I still think it's wild how short the contribution sounds when you say it out loud.

Echo Right.

Onyx Like 'we threw out recurrence and convolutions and just used attention' — that is the whole pitch.

Echo That's the pitch, but the reason it worked is the part people undersell. Before this, the dominant sequence models were RNNs, LSTMs, GRUs — and the fundamental constraint wasn't quality, it was sequentiality.

Onyx Mm-hm.

Echo An RNN factors computation along the symbol positions of the input. You compute hidden state h_t as a function of h_{t-1} and the input at position t. That inherently precludes parallelization within a training example. At longer sequence lengths, memory constraints limit how much you can batch across examples. So you're stuck paying a sequential cost that doesn't actually buy you anything except the ability to call it a recurrent model.

Onyx And the convolutional alternatives — ConvS2S, ByteNet — tried to fix the parallelism problem but the number of operations to relate two arbitrary positions still grew with distance. Linearly for ConvS2S, logarithmically for ByteNet. So learning long-range dependencies stayed hard.

Echo Exactly. And the Transformer cuts that to a constant number of operations regardless of distance. That's the lever. The whole claim isn't 'attention is elegant,' it's 'attention is constant-cost and parallelizable, and the quality holds.'

Onyx Which, honestly, explains why it won. Like — twelve hours on eight P100s to state of the art on translation? That's the sentence that made every lab with GPUs sit up.

Echo Yeah, and it's worth saying — attention wasn't new in this paper. Attention had been an integral part of compelling sequence models for a while. The contribution here is dropping everything else. To the authors' knowledge, this is the first transduction model relying entirely on self-attention, no sequence-aligned RNNs, no convolution.

Onyx Okay so walk me through the architecture, because I want to make sure I actually have the shape right.

Echo Sure. Standard encoder-decoder. Encoder maps input symbols x_1 through x_n to a sequence of continuous representations z. Decoder then generates output y_1 through y_m, one element at a time, auto-regressively — each step consumes the previously generated symbols.

Onyx Right.

Echo The encoder is a stack of six identical layers. Each layer has two sub-layers: a multi-head self-attention mechanism, and a position-wise fully connected feed-forward network. Residual connection around each sub-layer, then layer norm. So the output of each sub-layer is LayerNorm(x + Sublayer(x)). All sub-layers and embeddings produce outputs of dimension d_model equals 512 — that keeps the residual connections clean.

Onyx And the decoder?

Echo Also six layers, but with a third sub-layer that performs multi-head attention over the encoder stack's output. And the self-attention sub-layer is modified to mask future positions — combined with offsetting output embeddings by one position, that ensures predictions at position i can only depend on known outputs at positions less than i. Standard auto-regressive trick.

Onyx Okay, scaled dot-product attention. This is the part I want to make sure I have right.

Echo The framing is clean. An attention function maps a query and a set of key-value pairs to an output. You compute the dot product of the query with all keys, divide by sqrt(d_k), apply a softmax to get weights on the values, and the output is a weighted sum of the values. That's it.

Onyx Why divide by sqrt(d_k)?

Echo Honestly? Not much in the paper itself. The methodology is sound, the ablations are honest — they show that multi-head helps, that the scaling factor matters, that the number of heads is a real knob. The thing I'd flag is just that the paper is operating in a regime that looks almost quaint now: 512 dimensions, six layers, eight heads. Everything since has scaled these up by orders of magnitude and discovered new failure modes the original paper couldn't have seen.

Onyx Sure, but that's true of basically every foundational paper. You can only evaluate what you can build.

Echo Right. And the contribution holds. The constant-cost dependency modeling, the parallelization win, the encoder-decoder shape — all of that survived contact with scale.

Onyx Okay, this is the part where I have to be honest about something. I read this paper for the first time years after it came out, and the thing that struck me wasn't the math — it was how much of modern AI is downstream of one architectural choice. Like, GPT, BERT, T5, basically everything we've talked about on this show for nine months is a Transformer with knobs turned.

Echo That's not even an overstatement. The decoder stack gave you GPT-style autoregressive models. The encoder stack gave you BERT. The encoder-decoder stack gave you T5 and BART and most translation systems. Every modern frontier model is a Transformer with some combination of those pieces.

Onyx And the position-wise feed-forward — people forget that the FFN sub-layer is where most of the parameters live. The attention is the famous part, but the FFN is the bulk of the model's capacity.

Echo That's a genuinely good point and one the paper itself doesn't dwell on. Each layer's feed-forward is d_model times d_ff, with d_ff at 2048 — so the FFN has four times the parameters of the attention block. The attention is the routing; the FFN is the memory.

Onyx Okay I love that framing. The attention routes, the FFN stores.

Echo Yeah. And that has implications for everything from MoE architectures — where you basically make the FFN conditional — to interpretability, where a lot of the interesting structure lives in FFN neurons rather than attention patterns.

Onyx Hmm. So if you were designing the architecture today, knowing everything we know now, what would you do differently?

Echo Honestly? Probably not much at the architectural level. Maybe rotary position embeddings from the start, maybe pre-norm instead of post-norm — those are clean wins. But the core shape is right.

Onyx Yeah, that tracks. The paper got the shape right and the field has spent nine years tuning the details.

Echo And honestly, for a paper that's nine years old and still load-bearing — that's the strongest possible evidence the contribution was real.

Onyx Alright, this has been a fun one. Going back to the source material for once instead of talking about the twentieth paper that cites it.

Echo Yeah, no, I appreciated this. Sometimes you need to reread the thing itself.

Onyx Same time next week?

Echo Yeah, let's do it.