A. Smyntyna / Code
ENRU

bits per weight

The average number of bits each parameter actually takes up in a file, once the block scales and the promoted layers are counted. It is the number that sets the download, and it is never the number in the name: a Q4_K_M file averages 4.83 bits, not 4, so a 27B model lands at 16.3 GB rather than 13.5.

The arithmetic is worth doing once, because it is the only way to know what a listing costs before you press download. Parameters times bits per weight, divided by eight, gives gigabytes.

27 billion at 4.83 is 16.3 GB. The same 27 billion at 8.5, which is what Q8_0 comes out at, is 28.7 GB. The gap between those two is the entire quantization decision, and everything else is arguing about where in it to stand.

Two things push the average above the nominal bits. Every block of weights carries a scale factor stored alongside it, which is where Q8_0 finds its extra half bit. And a K-quant recipe promotes a handful of layer families to a higher depth, which is where Q4_K_M finds the other 0.83.

Read next