Looped Transformers in 2026: More Depth, Same Parameters

You hit a memory wall. Every extra layer in a standard model adds parameters that must sit in GPU memory. Looped transformers are one research answer: reuse the same layers instead of adding new ones.

Picture a common setup. An ed-tech team runs a coding-tutor app where students paste broken Python code and get the bug explained in English or Marathi. The setup is made up, but the problem is real.

The team serves its own small model on one rented GPU at about Rs 220 an hour. It has Rs 30 lakh (3 million rupees) for training, so every experiment must count.

The current model has 24 layers. A deeper one might explain bugs better, but the GPU must also hold working memory for 40 students at once. The team wants to know whether looping buys depth without that memory.

Four words carry this whole guide: layer, parameters, FLOPs and KV cache. Each one is a different budget, and looping moves only one of them.

Layer. A layer is one round of processing for every token. In a model that writes text, attention lets each token look back at earlier tokens. Then a feed-forward step, a small network applied to each token on its own, reshapes it.

Parameters. These are the numbers the model learns in training, also called weights. In a normal GPU setup, all of them sit in GPU memory while the model runs.

FLOPs. A FLOP, or floating-point operation, is one addition or multiplication on numbers with a fractional part, like 0.75. FLOPs per token is the work done for one token, which you pay for in GPU time and rupees.

KV cache. When a model writes text one token at a time, attention needs keys and values, two lists of numbers for every earlier token, in every layer. The KV cache saves them so they are not recomputed. It shares GPU memory with the parameters.

Looped transformers run one shared block of layers several times in a row, instead of stacking many different layers. The block's parameters are stored once, but each token passes through them on every loop.

Think of exam answer sheets. A stacked model hires three checkers, and each reads every sheet once. A looped model pays one checker to read every sheet three times.

Each checker is a block of layers, the salary is parameter memory and the reading time is compute. The salary bill drops to a third, but the reading time stays the same.

Where the picture breaks: three human checkers would all look for the same mistakes. Stacked layers each learn a different job, while a shared block must serve every loop with one set of weights.

The team sketches a looped option: 8 unique layers, run 3 times. Each token still gets 24 layer runs, but only a third of the layer parameters are stored.

Many real designs keep a few ordinary layers outside the loop. Papers call the layers before the loop the prelude and the layers after it the coda.

The 2025 recurrent-depth model below has a 2-layer prelude and a 2-layer coda. The September 2026 paper loops only the middle half of its layers.

A shared block costs memory once but compute every time it runs. The Python below counts both for the team's two designs, using rough textbook sizes rather than a real model.

Pause and predict: the looped design stores a third of the layers. What happens to its compute and its KV cache?

HIDDEN = 1024                            # toy: numbers per token
PARAMS_PER_LAYER = 12 * HIDDEN * HIDDEN  # attention 4x, feed-forward 8x
FLOPS_PER_PARAM = 2                      # one multiply, one add per weight

def design_cost(unique_layers, loops):
    params = unique_layers * PARAMS_PER_LAYER
    layer_runs = unique_layers * loops
    flops = layer_runs * PARAMS_PER_LAYER * FLOPS_PER_PARAM
    kv_slots = layer_runs                # one key-value slot per layer run
    return params, flops, kv_slots

for name, unique, loops in [("stacked", 24, 1), ("looped", 8, 3)]:
    params, flops, kv = design_cost(unique, loops)
    print(f"{name}: {params / 1e6:.0f}M params, "
          f"{flops / 1e9:.2f} GFLOPs per token, {kv} KV slots per token")

Run it with Python 3 and you should see two lines. The stacked line reads 302M params, 0.60 GFLOPs per token and 24 KV slots per token. The looped line reads 101M params, 0.60 GFLOPs and 24 KV slots.

What just happened? Both designs do the same 0.60 billion FLOPs per token and save the same 24 key-value slots. Only the parameters drop, from 302 million to 101 million.

The 12 x HIDDEN x HIDDEN size is the usual estimate for one layer whose feed-forward step is four times wider than HIDDEN. The 2 FLOPs per weight leave out attention's own work, which grows with text length.

Looping layers is an old idea with new results. A 2018 paper already reused one shared layer over repeated steps instead of stacking different layers. It also let each token position stop once a learned halting score said it was done.

In February 2025, Geiping and colleagues trained a 3.5 billion parameter recurrent-depth model on 800 billion tokens. Recurrent depth means the depth comes from repeating a block, not from adding layers.

That model has only 8 real layers: 2 before the loop, a 4-layer core block and 2 after. With 32 loops, its effective depth is 2 + 4 x 32 + 2 = 132 layers.

In training, the loop count was drawn at random, with a target average of 32. At test time, its scores on several reasoning tests rose as it ran more loops, with gains flattening at higher counts. It wrote no extra tokens to get there.

It found that looping still helped after the extra compute was taken away. The paper, arXiv 2609.01343, was first posted on 1 September 2026 and last revised on 11 September.

Earlier comparisons often matched only parameters, so the looped model quietly got more FLOPs. This paper matched FLOPs per token, parameters and KV cache, keeping each within about 4% of the baseline in its design table.

It uses a mixture-of-experts model, where a small router sends each token to a few of many small expert networks. The middle half of the layers runs twice.

To pay for the second visit, the authors made the layers narrower and added experts to win back parameters. Attention splits into parallel lookups called heads. Smaller heads, with each set of keys and values shared by more heads, kept the KV cache nearly flat.

The authors tested four sizes, the largest with 54 billion parameters outside the word lookup table. Loss is the model's error score in training, and lower is better.

On the compute-optimal frontier, the best result each training budget can buy, the looped model reached the same loss with 6.8% to 18.0% fewer training FLOPs. Its biggest gains on later tests were on code, and they grew with longer inputs.

The authors offer a partial reason. On the second visit, less attention lands on the attention sink, often the first token, which soaks up attention whatever the text says. More lands on useful tokens.

A 1 October 2026 paper, arXiv 2610.02185, turns the first loop's state into a weak guess and pushes the final guess away from it. It needs no extra training and no second model.

In plain terms, the final scores move further in the direction they changed since loop 1. If loop 1 and the final loop already agree on a word, little changes.

One variant applies this to the output scores and needs one extra output pass. The other applies it to the hidden numbers before the last layers and adds no output cost.

On a 2.6 billion parameter looped model, pass@1 on a 2024 maths contest set rose from 61.88% to 73.33%. Pass@1 is the share of problems solved on one try, here averaged over 16 samples per problem.

On the 2025 recurrent-depth model, pass@1 on a Python coding test rose from 22.56% to 31.71% with the hidden-number variant.

The authors also ran the models with half their loops, which alone lost 0.17 to 1.29 points on a seven-test multiple-choice average. They report that the method closed that gap in all six settings they tested, matching or beating full depth without it.

That half-depth setup used 22.5% to 48.2% fewer forward FLOPs, the work of running the model, not training it. The paper counts that arithmetic rather than timing real runs.

Yes, according to a vision-language paper, arXiv 2609.38426, posted on 29 September 2026 and revised on 13 October. Vision-language models read images and text together.

Its model has a language core of about 1 billion parameters, and in its default setting one forward pass, a single run through the model, makes 128 transformer-layer calls. The authors report it beats a range of similarly sized and larger non-looped models on image understanding and visual reasoning tests.

It does not win everywhere. In its main results table, six of 11 compared models beat it on chart reading. Seven beat it on a hallucination test, which checks for answers the image does not support. The authors also write that equal parameter storage is not the same as equal training FLOPs or wall-clock cost.

The tutor app may later read screenshots of error messages, so the team marks this paper as one to watch.

By default, looping shrinks only the weights. Compute, KV cache and latency, which is how long each answer takes, stay those of the full depth. The next section puts numbers on the cache.

Compute: every loop is real arithmetic. Triple the loops and the block's GPU time roughly triples too.

Latency: a stacked model also runs its layers in order, so at equal depth the sequential work is similar. Adding loops for extra depth adds waiting, which is why stopping early matters.

The recurrent-depth paper lets a token stop looping once two loops in a row give nearly the same prediction. The compute-matched and decoding papers measure cost in FLOPs rather than timing a serving setup, so measure latency yourself.

By default, as much as a stacked model of the same depth, because every loop runs attention and can save its own keys and values. For the team, that cache, not the weights, is the real squeeze.

The Python below sizes the cache for 40 students with 4,000 tokens each, using the toy model from earlier. Each key and each value is 1,024 numbers long, and a GiB is 1,073,741,824 bytes.

BYTES = 2                     # each number stored in 16 bits
KV_NUMBERS = 2 * 1024         # one key and one value per layer run
USERS, TOKENS = 40, 4000      # 40 students, 4,000 tokens each
GIB = 1024 ** 3

def kv_gib(full_runs, window_runs=0, window=64):
    per_run = KV_NUMBERS * BYTES
    full = full_runs * per_run * TOKENS
    recent = window_runs * per_run * min(window, TOKENS)
    return USERS * (full + recent) / GIB

print(f"stacked, 24 layers:      {kv_gib(24):.2f} GiB")
print(f"looped, every loop:      {kv_gib(24):.2f} GiB")
print(f"looped, one loop kept:   {kv_gib(8):.2f} GiB")
print(f"looped, loop 1 + window: {kv_gib(8, window_runs=16):.2f} GiB")
print(f"weights: {301_989_888 * BYTES / GIB:.2f} GiB stacked, "
      f"{100_663_296 * BYTES / GIB:.2f} GiB looped")

Run it and you should see 14.65 GiB for the stacked model and for the looped model that saves every loop. Keeping one loop's cache gives 4.88 GiB, and loop 1 plus 64-token windows gives 5.04 GiB.

The last line shows the weights: 0.56 GiB stacked against 0.19 GiB looped. Those sizes assume each number takes 16 bits, which is 2 bytes.

What just happened? In this toy, the cache for 40 students is about 26 times the stacked model's weights. Looping alone saves about 0.38 GiB of weights and nothing of the 14.65 GiB cache.

Capping the cache cuts it to about a third. Whether answers stay as good after that cut must be tested, model by model.

Real models often shrink every one of these numbers with grouped-query attention, where several query heads share one set of keys and values. The toy leaves that out to keep the arithmetic plain.

In the slot scheme, i mod k is the remainder of i divided by k. With k = 4, the fifth loop overwrites the first loop's keys and values.

The recurrent-depth paper tried that budget of 4 slots per token on a multi-turn chat test. The authors report that this did not reduce performance.

The looped coding model's serving code shows a more detailed design. Later loops read loop 1's full cache, keep their own cache of recent tokens and blend the two with a learned gate.

That code defaults to 2 loops and a 64-token window when the model's settings give neither. We read this in the source of two frameworks' releases from 16 and 18 October 2026, not in their docs. We did not check the model's own release notes. As of 19 October 2026, treat it as behaviour that can change, and pin your version. One of the two frameworks also dropped another looped model after its 25 July 2026 release.

Only as a measured pilot. The results come from research setups, and serving support covers one looped coding model in the open-source frameworks we checked. The team plans two weeks of tests before touching most of the Rs 30 lakh.

1. 1

   Match the budgets

   Compare designs on FLOPs per token, parameters and KV cache, as the compute-matched paper did. Write down any budget that differs.

2. 2

   Train two small twins

   Train the 24-layer model and the 8-layer block looped 3 times on the same data, capped at Rs 2 lakh.

3. 3

   Test on real tasks

   Score both on 500 real student bug reports, not only public tests.

4. 4

   Measure serving

   On the target GPU, record tokens per second and KV cache GiB with 40 students at once.

5. 5

   Check your serving stack

   Confirm your server can run the loop and keep a correct cache for each loop, or budget time to build it.

The twins match depth and FLOPs but not parameters, 302 million against 101 million, so the team writes that down. With a third of the parameters, the looped twin may score lower. Based on the sums above, the team expects the looped twin to save weight memory but not time. It expects the cache to still set the student limit. The team approves a larger test only if quality holds.

Frame it as a budget trade: weight memory against compute and KV cache per token.

1. 1

   Define it in one line

   A looped transformer runs one shared block several times, so depth grows while stored parameters stay fixed.

2. 2

   Separate the three budgets

   Name parameters, FLOPs per token and KV cache. Looping cuts the first, but not the other two by default.

3. 3

   Cite a fair comparison

   The September 2026 paper matched all three budgets and still reported training FLOP savings.

4. 4

   Handle the KV cache

   Cap cache slots per token, use smaller shared attention heads, or keep loop 1's cache plus short windows for later loops.

5. 5

   Talk about latency

   Layer runs go in order, so use early stopping or fewer loops for easy tokens.

6. 6

   State the risk

   Serving support is thin, so pilot it behind a flag before real users see it.