Mixture of Experts Explained: Routers, Active Parameters and VRAM

The model does not fit, because a mixture of experts model needs memory for all its weights, not just the ones each token uses. The small active number tells you the compute per token. The big total number tells you the memory.

Picture a common setup. A team runs a customer-support assistant for an online store and rents GPUs by the hour. They read that an open model has 26.3 billion total parameters but only 2.15 billion active, and they book a 24 GB GPU. The setup is made up, but the arithmetic below is real.

At 2 bytes per weight, 2.15 billion active weights is about 4.3 GB, so 24 GB looks generous. But the server must load all 26.3 billion weights, which is about 52.6 GB. The model fails to load before it answers a single question.

This guide explains why both numbers are true at once. We follow the same support assistant through routing, memory, training tricks and serving on several GPUs.

• Parameter or weight: one learned number inside the model. A 26.3 billion parameter model stores 26.3 billion such numbers.
• Layer: one stage of the model. A language model stacks many layers, and each one refines the numbers that stand for every token.
• Feed-forward block: the part of each layer that processes every token on its own. It multiplies the token by one big matrix, applies a simple function, then multiplies by a second matrix.
• Dense model: a model where every token goes through every weight in every layer.
• Router or gate: a small layer that scores the experts for one token and decides which ones run.
• Top-k: keep only the k highest scores. With top-2, each token goes to its two best-scoring experts.
• VRAM: the GPU's own memory. Weights must sit in VRAM, or be copied in, before the GPU can use them.

Mixture of experts replaces one feed-forward block with many copies, each with its own weights, plus a router that picks a few copies per token. The picked experts run, and their outputs are added together with the router's weights.

The original sparse layer paper defined the output as a weighted sum over experts. Experts with a gate value of zero are simply not computed. That skip is where the saving comes from.

In language models the experts take the place of the feed-forward block inside each Transformer layer, the standard layer design of today's language models. Some designs put experts in every feed-forward block, and others in every other one. Attention, the part where tokens look at each other, usually stays dense and shared.

The router multiplies the token vector by a small matrix to get one score per expert. It turns the scores into probabilities with softmax, keeps the top k, and uses those probabilities as weights.

Softmax is a function that turns any list of scores into positive numbers that add up to 1. The router matrix is tiny. In the made-up model used below, each token has 2,048 numbers and each layer has 32 experts. So the router has 2,048 x 32 = 65,536 weights per layer.

1. 1

   Score

   Multiply the token vector by the router matrix. You get one raw score, called a logit, for each expert.

2. 2

   Normalise

   Apply softmax so the scores become probabilities that add up to 1.

3. 3

   Keep the top k

   Keep the k largest probabilities and their expert numbers. With k = 2 out of 8 experts, six get nothing.

4. 4

   Renormalise

   Divide the k kept probabilities by their sum so they add up to 1 again. These become the weights for the weighted sum.

5. 5

   Run and combine

   Send the token to the k experts, multiply each output by its weight and add them up.

Total parameters are every weight the model stores. Active parameters are the weights one token actually uses on its way through. In a dense model the two are equal, and in a mixture of experts model active is much smaller.

Here is the support assistant's made-up model. Each token has 2,048 numbers, each expert's middle layer is 8,192 wide, and there are 32 experts per layer, top-2, over 24 layers. The vocabulary has 32,000 tokens.

Compute follows the active number. A matrix-vector product does about one multiply and one add per weight. So each token costs about 2 x 2.15 billion, or 4.3 billion operations, ignoring attention over earlier tokens.

Some designs use a gated feed-forward block with three matrices instead of two. Then each expert holds 3 x 2,048 x 8,192, about 50.3 million weights, and every expert row grows by half.

That table is the whole bargain. The top-1 routing paper used it to train a 1.6 trillion parameter model with 2,048 experts. A later paper trained a 269 billion parameter sparse model at roughly the compute cost of a 32 billion parameter dense one.

Because the router decides which experts a token needs only when the token reaches that layer. Different tokens in the same batch pick different experts, so once a step holds a few dozen tokens, almost every expert is in use. With 32 experts and top-2, it takes at least 16 tokens to touch them all.

We counted how many of the 8 experts a tiny test layer touched as the batch grew. Its code is in the build section below. The router there is untrained, so treat the exact numbers as an illustration. A trained router is pushed towards even spreading by the load-balancing loss covered later.

• 1 token: 2 of 8 experts used.
• 2 tokens: 3 of 8 experts used.
• 4 tokens: 5 of 8 experts used.
• 8, 32 or 512 tokens: all 8 experts used.

A serving engine batches many users together to keep the GPU busy. In the support assistant, that means dozens of tokens per step, so every expert is needed nearly every step. Leaving an expert off the GPU would make the whole step wait while it is copied in.

Some tools do keep cold experts in CPU memory or on SSD and copy them in on demand, which saves VRAM but makes steps wait for the copies. That is a separate topic. The default to plan for is all experts in VRAM.

Why MoE models can feel fast anyway

When one user generates one token at a time, each layer reads only the k experts that token picked. A survey of LLM inference shows that generating tokens at small batch sizes is limited by memory speed, not maths. So reading 4.3 GB instead of 52.6 GB per token can make a lone request quick.

With large batches the advantage shrinks for memory reads, because most experts get read each step. The compute saving per token stays. Shrinking the weights with lower precision helps memory directly: at 4 bits per weight the same model needs about 13.2 GB. You then have to test answer quality again.

You need a list of expert blocks, one linear layer for the router, and a loop that sends each expert only the tokens that picked it. The script below runs on a laptop CPU with PyTorch 2.14.1, the version we tested as of 23 October 2026. Install it with pip install torch numpy, since a missing NumPy prints a harmless warning.

import torch
import torch.nn as nn
import torch.nn.functional as F

torch.manual_seed(0)


class Expert(nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        self.up = nn.Linear(d_model, d_ff, bias=False)
        self.down = nn.Linear(d_ff, d_model, bias=False)

    def forward(self, x):
        return self.down(F.gelu(self.up(x)))


class TopKMoE(nn.Module):
    def __init__(self, d_model, d_ff, num_experts, k):
        super().__init__()
        self.k = k
        self.router = nn.Linear(d_model, num_experts, bias=False)
        self.experts = nn.ModuleList(
            Expert(d_model, d_ff) for _ in range(num_experts)
        )

    def forward(self, x):
        probs = F.softmax(self.router(x), dim=-1)    # one row per token
        top_w, top_idx = probs.topk(self.k, dim=-1)  # k experts per token
        top_w = top_w / top_w.sum(dim=-1, keepdim=True)
        out = torch.zeros_like(x)
        for e, expert in enumerate(self.experts):
            rows, slot = torch.where(top_idx == e)   # tokens sent to expert e
            if rows.numel() == 0:
                continue
            weight = top_w[rows, slot].unsqueeze(-1)
            out[rows] += weight * expert(x[rows])
        return out, probs, top_idx


moe = TopKMoE(d_model=64, d_ff=256, num_experts=8, k=2)
x = torch.randn(512, 64)                             # 512 tokens
out, probs, top_idx = moe(x)

per_expert = sum(p.numel() for p in moe.experts[0].parameters())
router = moe.router.weight.numel()
total = sum(p.numel() for p in moe.parameters())
active = router + moe.k * per_expert
print("output shape:", tuple(out.shape))
print("params per expert:", per_expert)
print("total params:", total)
print("active params per token:", active)
print("first token picks experts:", top_idx[0].tolist())
counts = torch.bincount(top_idx.flatten(), minlength=8)
print("tokens per expert:", counts.tolist())

On our run it printed these lines. The expert numbers depend on the random seed, so yours may differ on another version or machine. The parameter counts will not.

output shape: (512, 64)
params per expert: 32768
total params: 262656
active params per token: 66048
first token picks experts: [1, 4]
tokens per expert: [135, 119, 126, 127, 111, 147, 126, 133]

Check the numbers by hand. One expert has two matrices of 64 x 256, so 2 x 64 x 256 = 32,768 weights. Eight experts plus a 64 x 8 router give 262,656. One token touches the router and two experts, so 66,048, about a quarter of the layer.

The tokens-per-expert list adds up to 1,024, which is 512 tokens x 2 picks. That is the real cost of the layer: k expert runs per token, whatever the number of experts. We also compared this loop with a slow version that runs every expert with zero weights for unpicked ones, and the outputs matched exactly.

Softmax before or after the top-k step

Papers write the same idea in slightly different orders. The original sparse layer paper keeps the top-k logits first, then applies softmax. We checked that this gives the same weights as softmax first and renormalise after. The two differed by under 0.0000001 on 512 tokens.

Why top-1 routing cannot renormalise

With k = 1, renormalising makes the single weight exactly 1 for every token. The router then gets no learning signal from the model's output, apart from rounding noise. In our test the router's gradient size was 0.00004 with k = 1, against 91 with k = 2.

That is why the top-1 routing paper multiplies the chosen expert's output by the raw router probability instead. A gradient is the signal training uses to adjust a weight, so a zero gradient means the router never learns which expert suits a token. Only the extra balancing penalties covered below would still move it.

Noise and the second expert

The original paper added random noise, with a learned scale, to the scores before picking the top k, which spreads tokens more evenly. A later top-2 paper always used the best expert. It sent the token to the second expert only some of the time, with chance proportional to that expert's weight.

The original paper observed that, left alone, a router keeps favouring the same few experts. Those experts train faster and get picked even more. A load-balancing loss is an extra penalty added during training that grows when tokens pile onto a few experts.

The top-1 routing paper uses loss = alpha x N x sum of f_i x P_i over all N experts. Here f_i is the share of tokens sent to expert i, and P_i is the average router probability for expert i. It used alpha = 0.01.

Without alpha, the sum times N equals 1 when tokens spread evenly and rises towards N as they pile up. We applied the same formula to top-2, counting f_i as the share of the 1,024 routing slots. That extension is ours, so check your framework's exact version.

import math


def balance_loss(probs, top_idx, num_experts):
    counts = torch.bincount(top_idx.flatten(), minlength=num_experts).float()
    f = counts / counts.sum()      # share of routed slots per expert
    P = probs.mean(dim=0)          # average router probability per expert
    return num_experts * torch.sum(f * P)


print("balance loss:", round(float(balance_loss(probs.detach(), top_idx, 8)), 4))

for factor in (1.0, 1.25, 2.0):
    capacity = math.ceil(512 * moe.k / 8 * factor)
    dropped = int((counts - capacity).clamp(min=0).sum())
    print(f"capacity factor {factor}: capacity {capacity}, dropped {dropped}")

Add this below the first script and run it again. Our run printed a balance loss of 1.0016, close to the ideal 1.0. Every token going to one expert with top-1 would score 8.0, which is N.

A later paper added a second penalty, the router z-loss, which punishes very large router scores. It found this improved training stability and used a weight of 0.001.

Expert capacity is the most tokens one expert will take in a batch. The top-1 routing paper sets it to tokens per batch divided by experts, times a capacity factor. Tokens beyond that limit skip the expert.

A skipped token is not lost. It passes to the next layer through the residual connection, the shortcut that adds a layer's input to its output. It simply gets no expert update in that layer, which hurts quality if it happens often.

Fixed capacity gives every expert's work a known size in advance. The top-1 routing paper says its hardware required sizes declared ahead of time. It reports dropped rates typically under 1 percent and compared factors of 1.0, 1.25 and 2.0. A later paper trained at 1.25 and evaluated at 2.0.

To see why balance and capacity are linked, we pushed our tiny router towards expert 0 by adding a bonus to its score. Each batch has 512 tokens and 1,024 routing slots. The table shows dropped slots at three capacity factors.

With an even router, a factor of 1.25 drops nothing. With a skewed one, even doubling capacity still drops 17 to 25 percent of all slots. A bigger capacity factor buys some room, but only a balanced router fixes the problem.

Whether a serving engine drops tokens at inference time depends on the engine and its settings. Check its documentation rather than assuming the training rule applies.

Expert parallelism puts different experts on different GPUs, so each GPU stores only a share of them. Tokens then travel to whichever GPU holds their chosen expert and travel back after the expert runs.

The movement uses all-to-all communication. In one all-to-all step every GPU sends a different slice of its data to every other GPU at the same time. MoE needs two per layer: dispatch sends tokens out, and combine brings results home.

PyTorch exposes this step as torch.distributed.all_to_all_single. Each process splits its tensor into one chunk per process, and the chunks can have different sizes. We ran dispatch and combine on two CPU processes with gloo, PyTorch's CPU communication backend, and every token returned home in its original order.

In the support assistant, two GPUs of 40 GB each could hold the 52.6 GB of weights this way. Each holds about 27 GB, counting its copy of the shared attention weights, plus room for the KV cache. That split is made up for the example, so check your real model's numbers.

What it costs

All-to-all is network traffic in every MoE layer, twice. The original paper warned that experts need enough compute per byte moved, or the network becomes the bottleneck. The top-2 routing paper measured that all-to-all time grows slower than the number of devices, about 9 times for 128 times more.

Uneven routing also hurts more here. If one GPU's experts are popular, that GPU finishes last and every other GPU waits for it.

How vLLM does it, as of 23 October 2026

vLLM, an open-source LLM serving engine, turns this on with the --enable-expert-parallel flag. We read its 0.32.0 source, released on 18 October 2026. With the flag set, the source says each device owns a set of experts fully. Tensor parallelism splits each weight matrix across GPUs. Data parallelism runs full copies that each serve different requests, and each copy is called a rank. The docs give the expert-parallel size as tensor-parallel size x data-parallel size.

Its docs say attention weights are copied to every data-parallel rank when tensor parallelism is 1. An optional balancer, turned on with --enable-eplb, moves experts between GPUs and can add spare copies of experts. In 0.32.0 the spare count defaults to 0 and rebalancing runs every 3,000 steps by default.

MoE is the wrong choice when memory, not compute, is your tightest limit. It trades extra VRAM for cheaper tokens, so it pays off only when you can afford the memory.

• One small GPU: a dense model that fits will usually serve better than an MoE model that does not.
• Few, short requests: if the GPU sits idle most of the time, cheaper compute per token saves little.
• Weak links between GPUs: expert parallelism sends tokens across the network in every MoE layer.
• Fine-tuning without careful testing: one design paper names training instability and uncertain fine-tuning quality as the main things holding sparse models back.

Lead with the trade, then draw the layer, then size memory and compute separately. The strongest answers separate total from active parameters before anyone asks.

1. 1

   State the goal

   Say MoE grows a model's stored weights without growing the compute per token, by running only k of N experts.

2. 2

   Draw one layer

   Draw attention, then a router, then N feed-forward experts with top-k arrows, then the weighted sum and the residual connection.

3. 3

   Size it twice

   Work out total weights for VRAM and active weights for compute. Use the support assistant: 52.6 GB of weights, about 4.3 billion operations per token.

4. 4

   Name the training risks

   Mention router collapse, the load-balancing loss, and expert capacity with dropped tokens passing through the residual connection.

5. 5

   Plan the serving

   If weights exceed one GPU, use expert parallelism. Call out the two all-to-all hops per layer and the cost of busy experts.

6. 6

   Say what you would watch

   Track tokens per expert, dropped tokens if your engine drops any, per-GPU wait time, and VRAM headroom for the KV cache.