memory bandwidth
How fast the chip can read from RAM, in gigabytes per second. Generating text reads every active weight for every single token, so this, not raw compute, is what sets generation speed. An M4 Max moves about 546 GB/s; a desktop PC's system RAM is nearer 80 GB/s.
The estimate is blunt and close enough: divide bandwidth by the size of the weights you have to read per token. A 20 GB model on a 546 GB/s machine tops out around 27 tokens a second, and no amount of GPU is going to change that.
Two consequences. Shrinking a model makes it literally faster, not just smaller, because there is less to read per token. And a mixture-of-experts model is fast for the same reason: it only reads the experts the router picked.