The Agentic Post
Breaking
Perplexity vs ChatGPT Search vs Google AI Mode: Which AI Search Is Best in 2026?  Â·  What Are Vision-Language-Action (VLA) Models? How Robots Learn to Follow Instructions  Â·  How to Translate Documents With AI Accurately (and When Not To)  Â·  MCP vs A2A: The Two AI Agent Protocols Explained in Plain English  Â·  Is Grok Accurate for News? What Studies Show About AI Search Citations  Â·  Is AI a Bubble? 7 Warning Signs Investors Are Watching in 2026  ·  
Home/Guides/AI Glossary
What Is Quantization in AI? How Models Shrink to Run on Your Laptop

What Is Quantization in AI? How Models Shrink to Run on Your Laptop

AI Glossary

Quantization explained simply: how AI models store numbers with less precision to cut memory and cost, what 8-bit and 4-bit mean, GGUF files, and the quality trade-off.

Quantization is a way to shrink an AI model by storing its numbers with less precision, for example using 4 bits per number instead of 16. A 4-bit model takes roughly a quarter of the memory, runs faster and costs less, usually with only a small drop in quality. It is the main reason you can run capable open models on an ordinary laptop or phone.

How does quantization work, in simple terms?

An AI model is billions of numbers called parameters. Normally each is stored very precisely, like writing a price as 3.14159265 dollars. Quantization rounds it to something like 3.14. You lose a tiny bit of detail on each number, but across billions of them the model behaves almost the same while taking far less space.

What do 16-bit, 8-bit and 4-bit mean?

  • 16-bit (FP16 or BF16): the usual full-quality format; about 2 bytes per parameter
  • 8-bit (INT8 or FP8): about half the memory, with almost no quality loss for most tasks
  • 4-bit (INT4): about a quarter of the memory; the popular sweet spot for running models at home
  • Below 4-bit: very small files, but quality drops noticeably

Rough rule: a 7-billion-parameter model needs about 14 GB at 16-bit, 7 GB at 8-bit and around 4 GB at 4-bit, plus extra memory while it runs.

What is a GGUF file?

GGUF is a popular file format for quantized models, used by tools like llama.cpp, Ollama and LM Studio. When you download a model you will often see names like Q4_K_M or Q8_0; the number is roughly the bits per parameter. Higher is better quality and bigger. Our guide to running open-source models locally shows how to pick one.

Does quantization make AI models worse?

A little, and it depends on the task. Casual chat and summaries rarely suffer at 8-bit or good 4-bit. Precise tasks like maths, coding and long reasoning are more sensitive, so they benefit from higher precision. Bigger models usually handle quantization better than small ones, which is why a 4-bit large model often beats a full-precision small one.

Why do AI companies use quantization too?

To cut inference costs. Serving models at lower precision means more users per chip. New chips include fast support for 8-bit and even 4-bit maths for this reason. Combined with designs like mixture of experts, it is how giant models get affordable, as in our colibri guide where trillion-parameter models run in 4-bit.

Which quantization should I choose?

  • 8 GB of RAM: a small model (3 to 8 billion parameters) at 4-bit
  • 16 GB: up to roughly 14 billion parameters at 4-bit, or small models at 8-bit
  • 32 GB or more: mid-size models at 4-bit or 8-bit

Start with a Q4_K_M or similar, and step up if answers feel sloppy. Find more terms in our AI glossary.

Up Next
Claude Agent SDK vs OpenAI Agents SDK vs Google ADK: Which Should You Build With?

Claude Agent SDK vs OpenAI Agents SDK vs Google ADK: Which Should You Build With?

Agent Frameworks

The three big AI agent SDKs compared: Claude Agent SDK, OpenAI Agents SDK and Google ADK, on languages, model lock-in, tools, safety controls and best use cases.

If you are building an AI agent in 2026, the three big official toolkits are Anthropic’s Claude Agent SDK, the OpenAI Agents SDK and Google’s Agent Development Kit (ADK). Pick Claude’s for agents that work with files and code on a computer, OpenAI’s for fast prototypes, voice agents and freedom to switch models, and Google’s for large multi-agent systems on Google Cloud in several programming languages.

What is an agent SDK?

An SDK, or software development kit, is a ready-made set of building blocks. An agent SDK handles the repetitive parts of an agent: calling the model, running tools, keeping memory of the conversation, and passing work between agents. You add the instructions and tools for your use case. If agents are new to you, start with building your first AI agent.

How do the three SDKs compare?

Based on Composio’s hands-on comparison:

  • OpenAI Agents SDK: Python and TypeScript; works with OpenAI and over 100 other models; handoffs between agents, input and output guardrails, automatic session history, tracing, and realtime voice agents
  • Claude Agent SDK: Python and TypeScript; built for Claude, available through AWS Bedrock, Google Vertex AI and Microsoft Azure; built-in shell, file editing and search, subagents, MCP servers, permission modes and a spending cap per run
  • Google ADK: Python, TypeScript, Java and Go; tuned for Gemini but works with other models; sequential, parallel and loop agents, graph workflows, Agent2Agent (A2A) delegation and a web UI for tracing

Which agent SDK is best for beginners?

The OpenAI Agents SDK is usually the quickest to get running, with little setup and clear concepts. Google ADK is the most powerful for large systems but has more setup, including a required folder structure that trips up newcomers. The Claude Agent SDK is simple if your agent mainly needs to read, write and run things on a computer.

Does model lock-in matter?

Yes, for cost and resilience. The OpenAI SDK and Google ADK can swap models more easily, so you can move to a cheaper model later; see our AI API price comparison. The Claude Agent SDK is designed around Claude, which is a strength if Claude is your chosen model and a limit if you want to mix providers.

How do I keep an agent built with an SDK safe?

  • Give it only the tools it needs, with the narrowest permissions
  • Require human approval for actions that send, buy or delete
  • Set spending and step limits so loops cannot run up bills
  • Log every tool call so you can review what happened
  • Treat content from emails and websites as untrusted input

Real incidents, like the AgentCorruption attack and agents disabling monitors, show why these controls matter.

What about LangChain, CrewAI and other frameworks?

They remain popular, especially when you want independence from any one AI company. Our agent frameworks comparison and LangChain vs LangGraph cover them, and choosing an agent framework walks through the decision.