Mixture of Experts (MoE): How Today’s Biggest AI Models Work
How Mixture of Experts (MoE) makes AI models like GPT-4 and Mixtral more efficient — explained in plain English for UK developers and businesses.
GPT-4 is widely rumoured to have around 1.8 trillion parameters. If you ran all of them for every single word you typed, it would cost a small fortune and grind to a halt. So it doesn’t. Instead, it quietly routes each chunk of text to a small subset of specialised networks — and ignores the rest. That’s Mixture of Experts, and it’s the reason the most powerful AI models today are both enormous and practical at the same time.
Understanding MoE won’t tell you which stock to buy. But if you’re building with AI, deploying it in a product, or paying cloud bills for inference, it directly affects the choices you make and the costs you pay.
What Is a Mixture of Experts Model?
A Mixture of Experts model is a neural network architecture where the computation is split between many specialised sub-networks called experts. Rather than passing every input through all parameters every time, MoE uses a learned routing system to send each piece of text to only a small number of those experts. The rest stay idle for that particular token.
The concept isn’t new. MoE architectures were studied as far back as the 1990s by researchers including Michael Jordan and Robert Jacobs. What changed is scale. Modern large language models apply MoE specifically inside the feedforward layers of the transformer — the parts that do most of the heavy parameter lifting.
Think of it like a hospital with 20 specialist departments. When a patient comes in, they don’t visit every department — they go to cardiology, or neurology, or dermatology, depending on what’s wrong. The router decides which departments are relevant. The patient gets expert attention. The rest of the hospital keeps its capacity free. MoE works the same way, just at extraordinary speed and scale.
Dense vs Sparse: Why It Matters
A standard transformer — like the original GPT models or BERT — is a dense model. Every parameter participates in every forward pass. You double the parameters, you roughly double the compute required. That’s fine at small scale. At hundreds of billions of parameters, it becomes prohibitively expensive.
MoE models are sparse. They have a large total parameter count, but only a fraction of those parameters activate per token. Mixtral 8x7B, released by Mistral AI in December 2023, illustrates this clearly. It has 47 billion total parameters across eight expert networks. But for each token it processes, only the top two experts activate — meaning roughly 13 billion parameters fire at any one time. Inference cost is closer to a 13B model than a 47B one, while the model retains knowledge spread across all 47 billion parameters.
That gap between total capacity and active compute is where MoE earns its value. When I first worked through the numbers on Mixtral versus Llama 2 70B — roughly comparable in benchmark scores — the inference cost difference on cloud providers was stark. Mixtral was cheaper per token while matching or beating performance on most coding and reasoning tasks.
How the Router Actually Works
The gating mechanism is a learned weight matrix — a small neural network in its own right. For each token entering a feedforward block, the router produces a probability score over all available experts. It then selects the top-K scoring experts and sends the token to them. In most MoE implementations today, K equals 2: two experts handle each token, their outputs are weighted by the router scores, and the results are combined.
Top-2 routing is a deliberate design choice. Top-1 (winner-takes-all) is faster but unstable — the same expert tends to win repeatedly, leaving others undertrained. Top-2 adds a small cost but substantially improves training stability and expert specialisation.
There’s a catch. Left to itself, the router collapses. It discovers a few favourite experts and routes almost everything to them. The rest atrophy. To prevent this, researchers add an auxiliary load-balancing loss during training — a penalty that encourages the router to distribute tokens more evenly across experts. Getting that balance right is one of the harder engineering problems in MoE training. Too little balancing and you get expert collapse. Too much and the router ignores quality in favour of fairness.
Which Models Use MoE — and the Numbers Behind Them
Several of the most-used models today run on MoE architectures. GPT-4’s architecture has never been officially confirmed, but reporting from The Information in 2023 described it as using 16 expert networks with a top-2 routing scheme — suggesting roughly 220 billion active parameters from a total around 1.8 trillion. OpenAI hasn’t confirmed these figures.
What is confirmed: Mistral’s Mixtral 8x7B (47B total, 13B active), its successor Mixtral 8x22B (141B total, 39B active), and DeepSeek-V2 (236B total, 21B active). Google’s Gemini 1.5 Pro, which handles up to 1 million context tokens, uses a MoE architecture — and that long-context capability is partially enabled by MoE’s efficiency. DeepSeek-V3, released in late 2024, pushed further still: 671B total parameters with only 37B active per token, achieving performance competitive with Claude 3.5 Sonnet at a fraction of the compute cost.
UK developers using these models via APIs often don’t see the architecture directly — but they feel it. Mixtral via Fireworks AI or Together AI typically runs faster and costs less per million tokens than comparable dense models with similar benchmark scores. That matters when you’re building a product with real usage and real invoices.
The Efficiency Gains Are Real — With Caveats
The headline benefit of MoE is doing more with the same compute. A model can accumulate knowledge across 100 billion+ parameters but only activate 20–30 billion per token. Training is also more efficient — you can achieve strong benchmark scores at a given active-parameter budget by spreading total capacity across more experts.
There’s a subtler gain too: specialisation. Different experts do actually learn different things, though not in a cleanly organised way. Analysis of Mixtral’s expert routing showed that different experts activate preferentially for different token types — code versus prose, different languages, different syntactic roles. It’s not clean specialisation (experts don’t neatly divide by domain), but there is genuine differentiation. That’s part of why MoE models punch above their weight on diverse tasks.
When I looked at inference benchmarks across providers in early 2026, MoE models consistently offered better performance-per-pound on UK cloud pricing. On AWS Bedrock, for example, Mixtral 8x22B instruction was priced at roughly £0.003 per 1,000 input tokens — substantially below comparable dense models with similar reasoning scores. For businesses processing high volumes of text, those differences compound fast.
The Downsides Nobody Talks About
MoE models have a serious hardware problem: all expert weights must sit in memory simultaneously, even when most aren’t active. Mixtral 8x7B needs about 90GB of VRAM to run — considerably more than a standard 13B dense model’s 26GB footprint. You can’t skip loading the experts you won’t use, because you don’t know which you’ll need until the routing decision is made.
This creates a painful gap. MoE models are cheap per token but expensive to host. Running Mixtral 8x7B yourself requires at least two high-end GPUs. At 8x22B, you’re looking at four or more. For a UK startup or developer who wants to self-host rather than pay API fees, that capital cost is real and sometimes prohibitive.
Training is also harder. MoE introduces instability not present in dense models — the load balancing problem, routing collapse, and the challenge of keeping all experts meaningfully trained throughout a long run. Getting MoE training right at scale requires significant engineering resources. That’s why most well-functioning MoE models today come from well-resourced labs with large research teams, not from hobbyist fine-tuning projects. Load balancing failures during training can produce models that look fine on perplexity metrics but route poorly in practice — a subtle and frustrating bug to debug.
MoE vs Dense in Practice: Benchmarks Tell Part of the Story
On standard benchmarks — MMLU, HumanEval, GSM8K — top MoE models match or beat dense models with comparable active parameters. DeepSeek-V3 scored above 90% on MATH-500 and outperformed Claude 3.5 Sonnet on several coding benchmarks, using active parameters comparable to a strong 35–40B dense model.
But benchmarks don’t capture everything. MoE models can exhibit inconsistency on tasks that don’t cleanly match any expert’s training distribution. Some users report higher variance in output quality compared to dense models — excellent on common tasks, occasionally erratic on unusual ones. Whether that matters depends entirely on your use case.
For creative writing, summarisation, and code completion — tasks with clear training signal and large training data volumes — MoE models perform reliably. For highly specialised technical domains where training data is sparse, the routing may fall back on generalist experts and lose the efficiency gains. UK businesses in legal, medical, or regulatory sectors considering MoE-based systems should benchmark on domain-specific tasks, not just general benchmarks.
What This Means for You
If you’re a UK developer or business working with AI in 2026, MoE architecture affects three practical decisions. First, API pricing: MoE-based models (Mixtral, GPT-4 if the rumours are correct, Gemini 1.5) generally deliver strong performance per pound on cloud inference — worth factoring into provider comparisons. Second, self-hosting decisions: if you want to run a model locally or on your own infrastructure, a 47B-parameter MoE model is cheaper per token but more VRAM-hungry than a comparable dense model. Third, reliability: for high-volume production workloads on common tasks, top MoE models are genuinely competitive. For niche technical domains, test thoroughly before committing.
The broader story is that AI efficiency is increasingly a design choice, not just a hardware question. MoE is how the industry squares the circle of building more capable models without proportional compute cost increases. It’s messy, technically demanding to do well, and not always the right tool — but for the largest and most capable systems, it’s increasingly the standard approach.
This article is for educational purposes only and does not constitute financial advice. Cryptocurrency investments involve significant risk. Always do your own research.
Stay ahead of the market
Join our community of nearly 5,000 across YouTube, LinkedIn, X, and Facebook — weekly crypto, AI, and digital lifestyle insights every Thursday. No spam. Unsubscribe any time.
Partner picks
Build a smarter digital stack
Explore curated AI, automation, wealth, and creator tools selected for practical value, transparent pricing, and clear use cases.
Disclosure: some links may be affiliate links. DigitechLifestyle may earn a commission at no additional cost to you.
