decode
The writing phase. After prefill, the model produces one token per forward pass, each one appended to the prompt and fed back in. This is the phase tokens per second measures, and it is bandwidth-bound: every active weight has to be read out of memory for every single token.
Decode is strictly one token at a time, because the model cannot know what the second word is until it has committed to the first. That serial dependency is why extra GPU cores barely help here and why memory bandwidth does.
Two things follow. Shrinking a model makes decoding faster and not just smaller, because there is less to read per token. And the KV cache exists so that decode does not have to re-read the whole conversation for every word, which is the trade that turns a compute problem into a memory problem.
Because the chip spends most of decode waiting on memory, it has arithmetic to spare. Speculative decoding and multi-token prediction both exist to spend it.