A. Smyntyna / Code
ENRU

distillation

Training a small model using a big one as the teacher. Either the teacher is a file you own, in which case you can copy what it believes about every answer it did not give, or it is an API, in which case you only get the text it returned. Nearly every model small enough to run on your own machine is one or the other.

It takes real training and real money, which is why a frontier model gets pruned and quantized within days of release and distilled months later.

The full version of this, including what gets lost through an API and why the frontier keeps moving anyway, is its own post: Distillation: Racing to the frontier.

Read next

Elsewhere