Quantization is a way to shrink an AI model by storing its numbers with less precision, for example using 4 bits per number instead of 16. A 4-bit model takes roughly a quarter of the memory, runs faster and costs less, usually with only a small drop in quality. It is the main reason you can run capable open models on an ordinary laptop or phone.
How does quantization work, in simple terms?
An AI model is billions of numbers called parameters. Normally each is stored very precisely, like writing a price as 3.14159265 dollars. Quantization rounds it to something like 3.14. You lose a tiny bit of detail on each number, but across billions of them the model behaves almost the same while taking far less space.
What do 16-bit, 8-bit and 4-bit mean?
- 16-bit (FP16 or BF16): the usual full-quality format; about 2 bytes per parameter
- 8-bit (INT8 or FP8): about half the memory, with almost no quality loss for most tasks
- 4-bit (INT4): about a quarter of the memory; the popular sweet spot for running models at home
- Below 4-bit: very small files, but quality drops noticeably
Rough rule: a 7-billion-parameter model needs about 14 GB at 16-bit, 7 GB at 8-bit and around 4 GB at 4-bit, plus extra memory while it runs.
What is a GGUF file?
GGUF is a popular file format for quantized models, used by tools like llama.cpp, Ollama and LM Studio. When you download a model you will often see names like Q4_K_M or Q8_0; the number is roughly the bits per parameter. Higher is better quality and bigger. Our guide to running open-source models locally shows how to pick one.
Does quantization make AI models worse?
A little, and it depends on the task. Casual chat and summaries rarely suffer at 8-bit or good 4-bit. Precise tasks like maths, coding and long reasoning are more sensitive, so they benefit from higher precision. Bigger models usually handle quantization better than small ones, which is why a 4-bit large model often beats a full-precision small one.
Why do AI companies use quantization too?
To cut inference costs. Serving models at lower precision means more users per chip. New chips include fast support for 8-bit and even 4-bit maths for this reason. Combined with designs like mixture of experts, it is how giant models get affordable, as in our colibri guide where trillion-parameter models run in 4-bit.
Which quantization should I choose?
- 8 GB of RAM: a small model (3 to 8 billion parameters) at 4-bit
- 16 GB: up to roughly 14 billion parameters at 4-bit, or small models at 8-bit
- 32 GB or more: mid-size models at 4-bit or 8-bit
Start with a Q4_K_M or similar, and step up if answers feel sloppy. Find more terms in our AI glossary.




