Mixture of Experts — Why GPT-4 Is Smarter Than It Looks

Most modern frontier models (GPT-4, Mixtral, DeepSeek, Llama 4) are 'mixture of experts' (MoE). Instead of one giant brain, they have many smaller 'expert' brains. For each word generated, only 2–4 experts activate. The model has 1 trillion parameters total but only 'uses' 30 billion per word. Same quality, way faster.

Some examples

• Mixtral 8x7B: 8 experts, picks 2 per token. Looks like a 47B model, runs like a 13B.
• GPT-4 is widely believed to be MoE based on inference behavior (OpenAI hasn't confirmed).
• DeepSeek V3 has 671B total parameters but activates only 37B per token.
• Llama 4 went MoE in 2025 to keep up with this efficiency trick.

Try it!

Read the Mixtral paper's intro on Mistral's website. Notice the math: total params vs active params.

Mixture-of-Experts models like Mixtral or DeepSeek have huge total parameter counts but only activate a handful of 'expert' subnetworks per token. The result: capability close to a much larger model at the inference cost of a smaller one.

Some examples

• Mixtral 8x7B has 47B total params but uses only ~13B per token at inference.
• DeepSeek V3's MoE design lets it match GPT-4-class quality at a fraction of the serving cost.
• Mistral's MoE variants run faster than dense models of similar quality on the same GPU.
• Some Llama variants now use MoE to compete with closed models cost-effectively.

Try it!

Read the spec page for Mixtral or DeepSeek V3. Note the gap between total and active parameters. That's the trick.