tensor parallelism
Splitting each individual layer across machines, so every machine works on every token and their memory bandwidth adds up. It is the split that makes a cluster faster rather than only bigger. exo reports 1.8x on two devices and 3.2x on four, which is most of the way to linear and not all of it.
The gap between 4 machines and 3.2x is the interconnect. Every layer, the machines have to exchange partial results, so the network sits inside the inner loop rather than at the edges of it. That is why exo's fast path is RDMA over Thunderbolt 5 and why it wants every machine cabled to every other machine instead of daisy-chained.
Pipeline parallelism has the opposite profile: almost no communication, and no speedup either.