inference

Running a trained model to get output, as opposed to training it. When you type a prompt and the model generates a reply, that is inference. It splits into two phases: reading your prompt (prefill) and writing the answer one token at a time (decode).