KV cache
The model's short-term memory for the current conversation. As it reads your prompt it stores a key and value vector for every token so it does not have to re-read them for each new word. The cache grows with context length: roughly 1 to 4 GB for a 27B-class model at 32k tokens, and it lives in RAM alongside the weights.
The cache is the trade that makes generation affordable: a bit of memory spent so the model never has to re-read the conversation to write the next word. It grows in a straight line with the number of tokens, and it is the term that goes missing when people work out whether a model fits, because it is not in the model name and not in the download size.
The 1 to 4 GB range is that wide because the attention layout decides it. Grouped-query attention shares one key and value across several heads and cuts the cache by a factor of four to eight, so two models with the same parameter count can want very different amounts of room for the same chat.
It also does not creep up on you mid-conversation. You choose a context window when the model loads, and most runtimes reserve the whole cache right then, before a word is typed. The cost of a long chat is paid at load time, which makes context length the one setting that turns a model that fits into a model that does not.