Glossary

The jargon that gets in the way of local AI, in plain terms. Each entry links to the places it actually comes up, so you can read the idea and then go see it used.

56 terms · 0 used in the writing

active parametersHow much of a model actually runs to produce one token, as opposed to how much of it sits in memory. In a name like qwen3.6-35b-a3b the a3b is this number: 35 billion parameters stored, 3 billion of them used per token. It predicts speed, where the total predicts memory.
agentA model running in a loop with tools. It gets a goal instead of a question, asks to run something (read a file, run a command, search), gets the result back, and decides what to do next. The loop is what makes it an agent; the model itself only ever predicts the next token.
BF16Brain floating point, a 16-bit number format Google Brain designed for machine learning. It keeps the same 8-bit exponent as a full 32-bit float, so it reaches the same range of magnitudes, and pays for that by cutting the mantissa to 7 bits. Most open weights ship in BF16 at 2 bytes per parameter, which is why a 27B model is a 54 GB download.
bits per weightThe average number of bits each parameter actually takes up in a file, once the block scales and the promoted layers are counted. It is the number that sets the download, and it is never the number in the name: a Q4_K_M file averages 4.83 bits, not 4, so a 27B model lands at 16.3 GB rather than 13.5.
chat completionThe plainest way to use a model: you send a list of messages, it sends back one reply, and nothing else happens. No tools, no files, no second turn it decides on by itself. It is one HTTP request to something like http://127.0.0.1:1234/v1/chat/completions, and every agent and harness you will ever use is a program making that same call in a loop.
context windowThe most tokens a model can hold at once: system prompt, conversation, files it read, and everything it has said so far. Frontier models in 2026 quote 200k to 1M tokens. When it fills, something has to be dropped or summarized, and the model has no memory of what left.
decodeThe writing phase. After prefill, the model produces one token per forward pass, each one appended to the prompt and fed back in. This is the phase tokens per second measures, and it is bandwidth-bound: every active weight has to be read out of memory for every single token.
denseA model where every parameter runs for every token, with no routing. A 27B dense model does 27B parameters of work on each token it generates. Simpler than MoE, but slower at the same total size, because there is no way to skip most of the weights.
distillationTraining a small model using a big one as the teacher. Either the teacher is a file you own, in which case you can copy what it believes about every answer it did not give, or it is an API, in which case you only get the text it returned. Nearly every model small enough to run on your own machine is one or the other.
distributed inferenceRunning one model across several machines, because it does not fit on any of them. The memory pools and the model loads. The speed usually does not pool: exo reports 1.8x on two devices and 3.2x on four when it can shard the layers, and nothing at all when it can only chain them.
dynamic rangeThe distance between the largest and smallest value a format can hold. In a photograph it is the number of stops between the brightest highlight and the darkest shadow that still carry detail. In a number format it is set by the exponent bits, which is the whole reason BF16 exists.
edge deviceHardware that runs a model locally, out at the edge of the network, rather than calling a datacenter: a phone, a laptop, a small server in a closet. The point is that inference happens on the device itself, so the data never leaves. A 1-bit build of a 27B model, 3.9 GB on disk, runs on an iPhone 17 Pro Max at around 11 tokens per second.
exponentThe part of a floating-point number that sets its scale, separate from the mantissa that holds the digits. More exponent bits means a wider range of magnitudes before a number overflows to infinity or collapses to zero. BF16 spends 8 of its 16 bits here; FP16 spends 5, which is why FP16 training runs overflow and BF16 ones do not.
fine-tuneContinuing to train a released model on your own smaller dataset so it specializes. Costs a rounding error of what the original training cost, which is why one open-weights release turns into hundreds of variants for coding, documents, roleplay or a single company's tone.
FP16The older 16-bit float: 1 sign bit, 5 exponent bits, 10 mantissa bits. It holds finer detail than BF16 but reaches a much narrower range of magnitudes, so training runs in it can overflow. Both formats are 2 bytes per parameter, so they produce identically sized files.
GGUFThe single-file model format used by llama.cpp and the tools built on it, like Ollama and LM Studio. One .gguf file holds the weights, the quantization, and the metadata, so you download it and run. Recipe names like Q4_K_M describe how the layers were quantized inside it.
grouped-query attentionAn attention layout that shares one key and value across several query heads instead of giving every head its own. It shrinks the KV cache by a factor of four to eight with almost no quality cost, which is why a 27B-class model at 32k tokens wants 1 to 4 GB of cache rather than 10.
harnessThe program wrapped around the model that turns raw text prediction into something useful: it manages the prompt, runs tools the model asks for, feeds results back, and loops. The model only predicts tokens; the harness is what makes it an agent. Claude Code and similar tools are harnesses.
importance matrixA measurement of which weights a model actually leans on, made by running a corpus of real text through it and recording how much each weight moves the output. Quantizers use it to decide what can be crushed and what has to be protected. The IQ recipes in llama.cpp are named for it.
inferenceRunning a trained model to get output, as opposed to training it. When you type a prompt and the model generates a reply, that is inference. It splits into two phases: reading your prompt (prefill) and writing the answer one token at a time (decode).
just-in-time loadingLetting the runner load a model only when a request arrives and drop it again after an idle timeout, instead of holding the weights in memory all day. You trade a few seconds of load time per request for the gigabytes back in between. LM Studio does this per model, with the timeout as a setting.
K-quantllama.cpp's family of mixed-precision recipes, the ones with a K in the name. Weights are stored in blocks that share a scale factor, and a fixed rule promotes certain layer families above the nominal bit depth. The trailing S, M or L says how much promoting the recipe does.
KV cacheThe model's short-term memory for the current conversation. As it reads your prompt it stores a key and value vector for every token so it does not have to re-read them for each new word. The cache grows with context length: roughly 1 to 4 GB for a 27B-class model at 32k tokens, and it lives in RAM alongside the weights.
llama.cppThe C++ inference engine most local AI tools are built on, including Ollama and LM Studio. It runs on CPU, CUDA, Metal and Vulkan, which is why the GGUF files it reads work on almost any machine. The quant recipes with names like Q4_K_M and IQ3_XXS are its.
LM StudioA desktop app for running models locally: search a catalog, download a GGUF or MLX build, chat with it, and flip on a server that speaks the OpenAI-compatible API at http://127.0.0.1:1234. It is the version of this with no terminal in it, which is why most people start here.
localhostThe address a machine uses to talk to itself. `127.0.0.1`, and `localhost` is the name for it. A local model runner listens there, so `http://127.0.0.1:1234` reaches LM Studio on the machine you typed it on and nowhere else. Nothing sent to it leaves the computer or touches a network.
mantissaThe part of a floating-point number that holds its significant digits, separate from the exponent that sets the scale. In quantization, cutting mantissa bits is what actually loses precision: you keep the rough magnitude of each weight but round off the fine detail. It is the difference between storing 3.14159 and just 3.1.
MCP serverA small program that exposes tools or data to a model through the Model Context Protocol, a shared standard for how harnesses connect to tools. Instead of hard-coding an integration, you point the harness at an MCP server and the model can call whatever it offers: files, a database, an API. One protocol, many pluggable tools.
memory bandwidthHow fast the chip can read from RAM, in gigabytes per second. Generating text reads every active weight for every single token, so this, not raw compute, is what sets generation speed. An M4 Max moves about 546 GB/s; a desktop PC's system RAM is nearer 80 GB/s.
mixed precisionQuantizing different parts of a model to different bit depths instead of crushing all of it the same amount. A typical recipe holds the roughly 15% of load-bearing weights at 8 bits and pushes the other 85% down to 4, which costs a little more than uniform 4-bit and keeps most of what uniform 4-bit throws away.
MLXApple's array framework for running models on Apple Silicon, tuned to use the GPU and unified memory directly. MLX builds often run faster than the equivalent GGUF on a Mac and ship in their own format. It is the Apple-native alternative to the llama.cpp stack.
MoEMixture of Experts. The model is split into many small sub-networks (experts), and a router picks only a few to run for each token. A 35B model with 3B active per token stores all 35B in memory but does the math of a 3B model, so it answers much faster than a dense model of the same total size.
multi-token predictionSpeculative decoding with the draft model built in. The model is trained with an extra head that predicts the token after next, so it can propose its own continuations and verify them in one pass, with no second model to load or keep in memory. DeepSeek V3 popularized it.
multimodalA model that takes more than text as input. In practice this means images: screenshots, photos, scanned PDFs, encoded into the same token stream as the prompt. Some also handle audio. Output is still usually text.
MXFP4A 4-bit floating-point format that shares one scaling factor across a small block of weights, keeping more dynamic range than plain 4-bit integers. It is what GPT-OSS-120B and Kimi K3 ship in. The floating-point layout means the 16 available values are spread more usefully than evenly spaced integers.
OllamaA local model runner built on llama.cpp, driven from the terminal. `ollama run gemma4` downloads the weights and starts a chat. It also serves an OpenAI-compatible API on port 11434, which is what lets other software on your machine talk to the model without knowing anything about how it is loaded.
open weightsA model whose trained parameters are published as files you can download and run. Not the same as open source: the training data and the code that produced the weights are usually withheld. Llama, Qwen, Gemma, DeepSeek and gpt-oss are open weights. Fable and GPT are not.
OpenAI-compatible APIThe de facto standard shape for talking to a model over HTTP: POST a list of messages to /v1/chat/completions, get a reply back. OpenAI defined it, and every local runner copies it, so LM Studio and Ollama both answer on http://127.0.0.1:1234 or :11434 with the same request format.
parametersThe numbers inside the model, learned during training and frozen afterwards. A 27B model has 27 billion of them. Parameter count sets the file size directly: at 2 bytes each, 27 billion is 54 GB, and at 4-bit it is closer to 16 GB. It is the first number in almost every model name.
pipeline parallelismSplitting a model by layer across machines: the first machine holds layers 1 to 20, the next holds 21 to 40, and a token walks the chain. It is what makes an oversized model fit, and it makes nothing faster, because only one machine is computing at any moment.
prefillThe reading phase. Before a model writes anything it runs your whole prompt through itself, every token of it, filling the KV cache. This is where the wait before the first word comes from, and it is compute-bound: a dense 27B does 27 billion parameters of arithmetic per token you sent it, an A3B does 3 billion.
pruningDeleting parts of a trained model and keeping the rest, with no retraining. On a mixture-of-experts model that means dropping whole experts: Cerebras' REAP method cut Kimi K3 from 896 experts to 179, taking the file from 1.56 TB to 350 GB in two days. Cheap, fast, and it costs you capability that nothing puts back.
Q4_K_Mllama.cpp's most-downloaded 4-bit recipe, and the safe default. Q4 is four bits per weight, K means the K-quant scheme with per-block scales, and M is the medium size within it, where the attention and feed-forward layers that matter most get bumped above four bits. A 27B model lands near 16 GB.
quantizationStoring each weight in fewer bits to shrink the model. Full precision is 16 bits per weight; 4-bit quantization uses 4, cutting the file to about a quarter of its size. A 27B model drops from 54 GB at 16-bit to around 16 GB at 4-bit, with a small, usually acceptable, loss in quality.
reasoning modelA model trained to write out its working before it answers. The thinking pass is generated text like any other, so it costs time and tokens, and most interfaces hide it. On a measured M4 Max run a 35B A3B reached its first token in 2.66 seconds and then thought for 27 more.
routerThe small network inside a mixture of experts that decides, for every token, which few experts run. It is what makes a 35B model do the work of a 3B one: 35 billion parameters sit in memory and the router picks about 3 billion of them each time. Kimi K3 routes to 16 experts out of 896.
safetensorsThe plain weight-file format most labs release in, and what an MLX build actually is under the name. It stores tensors and a small header and nothing executable, which is the safe part: the older pickle-based format could run code when you loaded it.
speculative decodingA speed trick that costs no quality. A small draft model guesses the next few tokens, the big model checks all of them in one pass and keeps the ones it agrees with. The output is identical to running the big model alone, and the speedup is typically 1.5 to 2x, entirely determined by how often the small model guesses right.
system promptThe block of instructions the harness puts in front of your conversation before the model sees any of it: who it is, what it may do, what tools exist and how to ask for them. You do not type it, and it is charged against the context window like everything else.
tensor parallelismSplitting each individual layer across machines, so every machine works on every token and their memory bandwidth adds up. It is the split that makes a cluster faster rather than only bigger. exo reports 1.8x on two devices and 3.2x on four, which is most of the way to linear and not all of it.
tokenThe unit a model reads and writes. Not a word and not a character, but a fragment of one, produced by a lookup table built during training. English averages roughly four characters per token, so 1,000 words is about 1,300 tokens. Everything is priced, measured and capped in tokens: context windows, API bills, generation speed.
tokens per secondHow fast a model writes, and the number every benchmark leads with. Estimate it as memory bandwidth divided by the bytes of active parameters: 3 billion active params at 8 bits is 3 GB per token, so an M4 Max at 546 GB/s tops out near 180. Real measurements land at half to three quarters of that.
tool callingThe model writing a request for a command instead of an answer. The harness puts a list of available tools in the prompt, the model replies with text shaped like a call ("run bash with ls src"), and the harness runs it and pastes the output back as context. The model never executes anything itself.
TTFTTime To First Token: how long you wait after hitting enter before the model produces anything at all. It is dominated by prefill, the model reading your whole prompt, so a long prompt or document makes it worse. On a 27B dense model a large prompt can mean a minute or two of waiting before anything shows up.
unified memoryOne pool of RAM shared by the CPU and GPU, as on Apple Silicon, instead of separate system RAM and dedicated video RAM. It means a Mac with 128 GB can hand almost all of it to a model, with no copying between chips. macOS caps the GPU-usable share at roughly 65 to 75% of total.
VRAMThe memory soldered to a graphics card, separate from the computer's system RAM. It is the hard ceiling for local models on a PC: consumer NVIDIA cards ship with 8 to 24 GB, and the high end reaches 32 GB. A model that does not fit spills into system RAM and slows down by an order of magnitude.