Quantisation Explained: How AI Models Get Smaller Without Getting Dumber

Every time a new AI model launches, headlines chase the biggest, most powerful version. But the quieter story in 2026 is the opposite direction: shrinking these giant models down so they run on a laptop, a phone, even a smart speaker, without losing much of what made them useful in the first place. The technique behind that shrinking is called quantisation, and UK developers keep asking about it because it’s the difference between an AI feature that costs a fortune to run and one that doesn’t.

What Quantisation Actually Means

At its core, a neural network is a huge pile of numbers — weights — that determine how the model responds to input. Those numbers are usually stored with high precision, often as 32-bit or 16-bit floating point values. Quantisation reduces that precision, packing the same weights into 8-bit, 4-bit, or even smaller representations.

Picture a photograph saved at full resolution versus a compressed JPEG. The compressed version takes up a fraction of the space. Look closely and you might spot some quality loss. From a normal viewing distance, it’s often impossible to tell the difference. Quantised AI models work on roughly the same principle, trading a small amount of precision for a large amount of efficiency.

The maths matters here. Cutting a model from 16-bit to 4-bit precision doesn’t just save a bit of space — it can shrink the model’s memory footprint by roughly 75%, sometimes more. That’s the gap between a model that needs a data-centre GPU and one that runs comfortably on a mid-range laptop.

Why Bigger Isn’t Automatically Better

There’s a persistent assumption that more parameters always means a smarter model. It’s not wrong, exactly — but it misses the practical reality most developers face. A 70-billion-parameter model running at full precision might be marginally more capable than its quantised 8-bit cousin, but if it takes ten times the hardware and costs ten times as much to run, “marginally more capable” rarely wins the business case.

Quantised models also load faster and respond quicker, because there’s simply less data to move through memory. For anything customer-facing — a chatbot, a search assistant, a voice interface — that speed difference is the gap between a snappy tool people actually use and a sluggish one they abandon.

Falls apart fast if you push quantisation too aggressively, though. Below a certain precision threshold, usually around 2-bit, models start producing genuinely broken output — repeating themselves, losing track of context, or generating nonsense. There’s a sweet spot, and finding it is part science, part trial and error.

How Quantisation Works Under the Hood

Two broad approaches dominate. Post-training quantisation takes a model that’s already fully trained and compresses it afterwards — fast and cheap, but can lose more accuracy. Quantisation-aware training builds the precision reduction into the training process itself, so the model learns to compensate for the lower precision as it goes. Slower and more expensive to produce, but the results usually hold up better.

There’s also a granularity choice. Some methods quantise an entire model uniformly. Others quantise different layers at different precisions — keeping the most sensitive layers at higher precision while compressing the rest harder. This mixed-precision approach tends to preserve quality better than a blunt, one-size-fits-all cut.

Tools like GPTQ, AWQ, and GGUF have become the standard formats developers reach for. Each makes different trade-offs between compression ratio, speed, and how much accuracy survives the process — and picking the right one has become its own small area of expertise.

Where This Shows Up in Everyday Tech

UK users encounter quantised models constantly without realising it. On-device features on modern smartphones — live translation, photo search, voice transcription — almost always run quantised models locally, because sending every request to a cloud server would be slow, expensive, and a privacy headache.

Open-source AI communities have leaned into this hard. Models that would otherwise need enterprise-grade hardware now run on consumer laptops thanks to aggressive quantisation, letting hobbyists and small UK startups experiment with capable AI without a five-figure cloud bill.

Even large AI providers use quantisation internally to cut serving costs. When a company handles billions of requests a day, shaving even a small percentage off compute costs per request adds up to genuinely significant savings — savings that often get passed on as cheaper API pricing for developers.

The Trade-Offs Nobody Puts on the Marketing Slide

Quantisation isn’t a free lunch. Certain tasks are more sensitive to precision loss than others. Maths-heavy reasoning tends to degrade faster under aggressive quantisation than simple conversational chat, because small numerical errors compound across multi-step calculations.

Benchmarks can also mislead. A quantised model might score nearly identically to its full-precision original on standard tests, while quietly performing worse on edge cases the benchmark never covers. UK teams building anything safety-critical — medical, legal, financial — need to test quantised models against their own real use cases, not just published benchmark scores.

There’s a hardware dependency too. Not every chip handles low-precision arithmetic efficiently. Running a 4-bit model on hardware that isn’t optimised for it can sometimes be slower than running a well-optimised 8-bit version, which catches teams out when they assume “lower precision” always means “faster.”

What to Ask Before You Deploy a Quantised Model

Start with your actual constraint. If you’re memory-limited — running on a phone, a browser, edge hardware — aggressive quantisation is usually worth the accuracy trade-off. If you’ve got server-grade hardware and accuracy is everything, lighter quantisation or none at all might make more sense.

Test against your specific workload before committing. Generic benchmarks won’t tell you how a quantised model handles your customer service transcripts or your product catalogue. Run your own evaluation set through both the full-precision and quantised versions and compare the gap directly.

Watch the ecosystem too — quantisation techniques are improving quickly, and what required a noticeable accuracy trade-off a year ago often needs almost none today. A method that felt experimental in 2025 is standard practice by mid-2026.

Common Quantisation Formats Compared

GGUF has become the go-to format for running models on ordinary consumer hardware, popularised by the llama.cpp project. It supports a range of bit-widths in the same file format, from a relatively gentle 8-bit down to an aggressive 2-bit, letting developers pick their own trade-off point without switching tooling.

GPTQ takes a different approach, optimising specifically for GPU inference. It tends to hold up well at 4-bit precision and became popular early because it was one of the first methods to make 4-bit quantisation genuinely usable rather than just theoretically possible.

AWQ — activation-aware weight quantisation — looks at which weights actually matter most for a model’s outputs and protects those from compression more carefully than the rest. In practice, this often means better accuracy retention than a uniform quantisation approach at the same bit-width, at the cost of a slightly more complex setup process.

None of these formats is universally “best.” The right pick depends on your hardware, your framework, and how much engineering time you want to spend tuning things versus just getting something working today.

A Quick Example: The Same Model, Three Ways

Take a hypothetical 7-billion-parameter open model. At full 16-bit precision, it needs roughly 14GB of memory just to load — out of reach for most laptops. Quantised to 8-bit, that drops to around 7GB, workable on a decent gaming laptop. Pushed to 4-bit, it shrinks further to around 3.5 to 4GB, small enough to run comfortably alongside other applications on an ordinary machine.

The catch, predictably, is quality. Independent testing on reasoning-heavy tasks often shows the 8-bit version performing within a percentage point or two of the full-precision original, while the 4-bit version shows a slightly larger, though usually still modest, gap. For most everyday tasks — summarising, drafting, simple Q&A — the difference is barely noticeable to a human reader. For tight logical reasoning or precise maths, it becomes more visible.

This is exactly why “quantised” isn’t a single answer. A customer support chatbot and a financial modelling tool have wildly different tolerance for that gap, even if they’re built on the same underlying model.

What This Means for You

For UK businesses evaluating AI tools, quantisation is one of the reasons costs are falling even as capability rises. It’s a big part of why AI features that once needed expensive cloud infrastructure now run cheaply, sometimes even offline. When a vendor pitches an AI feature as fast and affordable, quantisation is very often part of the reason why.

If you’re building anything yourself, don’t treat quantisation as an afterthought bolted on at the end. Planning for it early — choosing a model family with strong quantised versions available — can be the difference between a project that scales affordably and one that gets stuck with an unsustainable cloud bill.

I’ve seen this pattern with three different open-source projects now: teams that ignore quantisation until launch week end up scrambling for cloud budget they didn’t plan for. Teams that bake it into their architecture from day one ship cheaper, faster, and with far fewer surprises on the invoice.

This article is for educational purposes only and does not constitute financial advice. Cryptocurrency investments involve significant risk. Always do your own research.