CPU vs GPU vs TPU for AI: Which Chip for Which Job

You pay for chips that sit idle, or you wait for work that a better-matched chip would finish sooner. The gpu vs tpu vs cpu question has a different answer for each job, not one answer for the whole company.

Picture a common setup. A search team builds meaning-based search for Indian languages such as Telugu, Hindi, Tamil and Kannada. The setup is made up, but the problem is real.

The product turns text into embeddings: lists of numbers that capture meaning, so similar sentences get similar numbers. The team runs three kinds of AI work on one embedding model.

• Monthly fine-tune. The team trains the model further on 5 crore (50 million) sentence pairs, so it handles its languages better.
• Re-embed. When the model changes, all 40 crore documents need fresh embeddings, because vectors from old and new models cannot be mixed.
• Live queries. Each user's search text becomes an embedding, about 300 times a second at the evening peak.

Everything runs on rented GPU servers. The monthly cloud bill has reached Rs 38 lakh (3.8 million rupees), and the usage graphs show three problems.

• The live query GPUs are busy at 7 p.m. and nearly idle at 3 a.m. Across the day, they do useful work about 5 percent of the time.
• The monthly training run takes four days, and nothing ships until it finishes.
• The re-embed blocks the training servers for a week every time the model changes.

The team picked GPUs once, for training, and never asked again for the other two jobs. This guide works out which chip fits each job, and why.

A CPU has from a handful to about 200 strong cores built for varied, step-by-step work. A GPU has thousands of simple arithmetic units doing the same step on many numbers. A TPU is built around large fixed grids that only do matrix maths.

The CPU: fewer, stronger cores

The central processing unit is the main chip in every server. A core is an independent worker inside the chip. AMD's EPYC 9005 server CPUs, for example, have up to 192 cores.

Each core is good at branches (if this, do that), odd jobs and constant switching between tasks. It also has vector units that apply one step to a short row of numbers at once. Some recent server CPUs add small matrix units too, such as Intel AMX on 4th Gen Xeon chips.

The GPU: thousands of simple units

The graphics processing unit was built to colour millions of screen pixels at once. NVIDIA's H100 SXM5 has 132 streaming multiprocessors, which are groups of simple cores, with 16,896 32-bit arithmetic units, which NVIDIA calls CUDA cores, in total.

It also has 528 Tensor Cores, units built only for small matrix multiplications. So a modern data-centre GPU has matrix units too. It keeps them inside a very flexible, programmable chip.

The TPU: fixed grids for matrix maths

The tensor processing unit is Google's custom chip for neural network maths. A tensor is a grid of numbers with any number of dimensions. Each TPU chip has one or more TensorCores, and each TensorCore holds matrix-multiply units, called MXUs, plus vector and scalar units.

An MXU is a systolic array: a fixed grid of multiply-and-add cells that pass numbers to their neighbours. Google's TPU architecture page lists 128 by 128 cells before v6e, and 256 by 256 on v6e and TPU7x.

As of 19 October 2026, the open way to get Google's data-centre TPUs is to rent them through Google Cloud. In April 2026, Google said it would also begin delivering TPUs to a select group of customers for their own data centres. In July 2026, it said the first such deliveries took place in the April to June quarter. Code for them must be compiled by XLA, a compiler that turns the model's maths into a fixed plan for the chip.

A picture to hold on to

Think of a car wash. Each part of the chip story maps to one part of the wash.

• CPU: a few skilled detailers. Each can handle any car, a scratch or an odd request, but the shop has far fewer of them.
• GPU: a huge wash bay with thousands of washers following the same instruction, plus some machines for the standard wash.
• TPU: an automatic tunnel wash. Cars roll through fixed brushes in a set order, very cheaply per car, as long as each car is a standard size.
• Memory: the road to the gate. However fast the washers are, they only wash cars that have arrived.

Where the picture breaks: a chip does not give one request to one worker. It splits one big multiplication across thousands of units, so even a single request can use the whole chip.

Nearly every step in a neural network multiplies a grid of inputs by a grid of learned weights. A model is mostly stacked layers of these multiplications. So the chip that multiplies fastest, and is fed numbers fastest, runs the model fastest.

Weights are the numbers a model learns during training. The team's embedding model has about 30 crore of them. A query is split into tokens, small word pieces, and each token's numbers pass through layer after layer of multiply and add.

Each multiply is tiny. The trouble is volume: each token needs about 2 operations per weight, so a 10-token query needs roughly 6 billion, before the smaller attention maths. Within one layer they do not depend on each other, so thousands of units can share them.

The first TPU took this furthest. Google has used it in its data centres since 2015, and its 2017 paper describes a matrix unit of 65,536 8-bit multiply-and-add cells. It was built for inference, the phase where a trained model answers requests.

Because a chip can only multiply numbers that have reached it. A job that does little maths per number fetched waits on memory, however many cores the chip has.

Memory bandwidth is how many bytes per second a chip can read from its memory. Arithmetic intensity is how many maths operations a job does for each byte it reads. The roofline model, from Williams, Waterman and Patterson at UC Berkeley, links the two.

Its rule: attainable speed is the smaller of the chip's peak maths speed and memory bandwidth times arithmetic intensity. The ridge point is the intensity where the two limits meet. Below it a job is memory-bound; above it, compute-bound.

Back at the car wash, the tunnel can wash 300 cars an hour. If only 20 cars an hour reach the gate, it washes 20. The brushes are not the limit; the road is.

The easiest way to raise arithmetic intensity is batching: processing many inputs together. The weights are fetched once and reused for every input in the batch.

It is far below: about 1 operation per byte. The chip spends almost all its time waiting for weights. The diagram shows how batching moves one layer up towards the ridge.

Neither wins everywhere. A GPU is the safer default because almost all AI code runs on it. A TPU can cost less per result on large, steady jobs whose shapes stay fixed, and only your own benchmark can show it.

Google's Cloud TPU guide lists good fits. These include models with no custom operations in the main training loop and models that train for weeks or months. Large models with large effective batch sizes are on it too, and so are models with ultra-large embeddings, common in ranking and recommendation.

The same guide lists poor fits. These include programs with frequent branching or many element-wise operations, maths applied to each number separately. Work that needs high-precision arithmetic and networks with custom operations in the main training loop are also on it.

Shapes matter too. XLA compiles its plan for the exact sizes of the inputs, and text is the common trouble, since every sentence has a different length. The usual fix is padding: rounding every input up to one of a few fixed lengths.

Where TPUs are strong

• Very large training runs spread over many chips. The TPU v4 paper describes a 4,096-chip system whose optical circuit switches can change how the chips connect.
• Long, steady jobs with fixed shapes, where one compiled plan runs for hours or days.
• Teams already on Google Cloud, with code that XLA compiles well.

Where GPUs are strong

• Almost every AI framework, library and tutorial supports them.
• Models with custom operations or shapes that keep changing, the cases Google's own TPU guide steers away from TPUs.
• Choice of supplier: you can rent them from many clouds, or buy and run your own.

Here is the decision the team sketches. Answer it once for each workload, starting at the top.

Training usually runs on GPUs or TPUs, because it repeats huge batches of matrix maths for hours or days. CPUs fit only small, simple models; Google's TPU guide also lists simple models that train quickly as a CPU job.

Training means showing the model examples and nudging its weights after each batch. Each step runs the model forward, measures the error, then runs backward to compute gradients: how much each weight should change.

For a dense layer, the backward pass does two matrix multiplications the same size as the forward one. One gives the gradient for the inputs, the other for the weights. So backward needs about twice the maths of forward.

Training also needs a lot of memory. The chip holds the weights, the gradients, the optimizer's extra numbers and the in-between results of every layer. The optimizer is the rule that turns gradients into weight updates.

A common trick is mixed precision training: most maths in 16-bit numbers, with a 32-bit master copy of the weights. The 2017 mixed precision paper estimates this cuts memory use by nearly 2x.

The team's monthly fine-tune uses padded fixed-length inputs, standard layers and no custom operations. That makes it a fair TPU candidate. They run the same job for one hour on a GPU server and on a TPU slice, a rented group of TPU chips.

They count training steps finished in that hour, not the peak speed on the spec sheet.

Use a GPU when steady traffic fills large batches, or when the model is too big to read quickly from CPU memory. Use a CPU when the model is small and traffic is light or spiky.

Inference means using a trained model to answer a request, such as one embedding for one search query. It runs only forward, with no gradients. Latency, how long one request takes, matters most here.

A live query arrives alone. To fill a GPU, a server can wait a few milliseconds and group queries that arrive close together. This is dynamic batching; NVIDIA's Triton server, for example, waits up to a delay you configure.

Dynamic batching helps at 7 p.m. At 3 a.m. there is nobody to group with, so the GPU waits on memory or sits idle.

The team's query model is small, about 30 crore weights. Stored as 8-bit numbers that is 300 MB (286 MiB), which fits easily in a CPU server's memory. A test gives about 15 milliseconds per query on 4 CPU cores (made up).

That is inside the 50 millisecond target. One 32-core server can run 8 such queries side by side, about 530 a second. So two CPU servers cover the 300 a second peak, with room for one to fail.

Large language models are different. A model with 70 billion weights needs 140 GB for its weights alone in 16-bit numbers. For one user, a dense model, one that uses every weight for every token, reads all its weights again for each new token.

For a single user, memory speed rather than maths sets the pace. That is why chips with very fast memory matter so much for these models.

It depends on whether you embed a big batch or live queries. Backfills belong on GPUs or TPUs fed large batches. Live query embeddings from a small model often run fine and cheaply on CPUs.

The 40 crore document re-embed is the perfect batch job. Nobody waits on any single document. The team can sort documents by length, pad them into a few fixed sizes and send huge batches, so the chip stays busy for the whole run.

Utilisation is the share of rented time a chip spends on useful work. A chip at 5 percent utilisation costs twenty times more per useful result than the same chip fully busy. That is the live query GPUs' real problem.

Divide the layer's maths by the bytes it moves, then compare that with the chip's ridge point: peak maths speed divided by memory bandwidth. Real chips need more reuse than most people expect.

Sparsity is a special mode that skips some zero weights. NVIDIA's Hopper architecture post lists the sparse figure as double the dense one, so ordinary dense work gets half the headline number.

Units matter too. Google lists v5e bandwidth in GiB per second, where 1 GiB is 2 to the power 30 bytes. Reading it as 800 GB per second would overstate the ridge point by about 7 percent.

The code below works out the intensity of one 4096 by 4096 layer, in 16-bit numbers, for four batch sizes. It then checks the result against both ridge points.

def matmul_intensity(batch, k, n, bytes_per_num=2):
    flops = 2 * batch * k * n                  # one multiply + one add
    moved = bytes_per_num * (batch * k + k * n + batch * n)
    return flops / moved                       # FLOPs per byte moved

RIDGES = {"H100": 1979e12 / 2 / 3.35e12,       # dense = half the sparse figure
          "TPU v5e": 197e12 / (800 * 2**30)}   # 800 GiB per second

for batch in (1, 64, 256, 512):
    i = matmul_intensity(batch, 4096, 4096)
    busy = [chip for chip, r in RIDGES.items() if i >= r] or "none"
    print(f"batch {batch:>3}: {i:5.1f} FLOPs per byte, compute-bound on: {busy}")

The moved line counts inputs, weights and outputs at 2 bytes per number. The weights term, k times n, does not grow with the batch, while the maths does. Run it and you should see these four lines.

• batch 1: 1.0 FLOPs per byte, compute-bound on: none
• batch 64: 62.1 FLOPs per byte, compute-bound on: none
• batch 256: 227.6 FLOPs per byte, compute-bound on: none
• batch 512: 409.6 FLOPs per byte, compute-bound on: ['H100', 'TPU v5e']

What just happened? For a wide layer and a small batch, the intensity is roughly the batch size. So this 16-bit layer needs a batch of a few hundred rows before either chip stops waiting on memory.

This is a rough model. It assumes every number crosses memory once, and it ignores on-chip caches, attention and other operations. Use a profiler to measure the real job before trusting it.

See the effect on your own CPU

You can watch the same effect on a laptop with NumPy. This script multiplies one 64 MiB weight matrix by a batch of 1 and a batch of 256, and prints the speed in GFLOP/s, billions of operations per second.

import time
import numpy as np

k = n = 4096
W = np.random.default_rng(0).standard_normal((k, n), dtype=np.float32)

for batch in (1, 256):
    x = np.ones((batch, k), dtype=np.float32)
    reps = 200 if batch == 1 else 20
    x @ W                                      # warm-up call
    start = time.perf_counter()
    for _ in range(reps):
        x @ W
    seconds = (time.perf_counter() - start) / reps
    print(f"batch {batch:>3}: {2 * batch * k * n / seconds / 1e9:.1f} GFLOP/s")

We ran it 11 times on a shared 2-core test machine with NumPy 2.3.4. Batch 1 gave 1.2 to 4.3 GFLOP/s and batch 256 gave 18 to 70 GFLOP/s. Batch 256 was 7 to 30 times faster per operation in each run.

Your numbers will differ with your CPU and its maths library. Expect the batch of 256 to be many times faster per operation.

float16 cannot store very small numbers, so tiny gradients round to zero, and loss scaling multiplies them up first. bfloat16 has the same range as 32-bit floats, so gradients this small do not underflow. It loses precision instead.

Both formats use 16 bits. float16 has 5 exponent bits for range and 10 fraction bits for precision. bfloat16 keeps float32's 8 exponent bits and has only 7 fraction bits.

Google's bfloat16 page says the dynamic range of bfloat16 and float32 are equivalent. By default, TPUs multiply bfloat16 values and add up the results as float32.

Here is a check you can run with NumPy and the ml_dtypes package, which adds bfloat16 to NumPy. We ran it on ml_dtypes 0.5.3.

import numpy as np
import ml_dtypes

bf16 = ml_dtypes.bfloat16
g = 1e-8                                       # a tiny gradient value

print(np.float16(g))                           # float16
print(np.array(g, dtype=bf16))                 # bfloat16
print(np.float32(np.float16(g * 1024)) / 1024) # float16 with loss scale 1024
print(np.array(1 + 1 / 256, dtype=bf16))       # a small update in bfloat16

It prints 0.0, then 1.00117e-08, then 1.0011718e-08, then 1. Here is what each line shows.

• float16 rounds 1e-8 to zero, because its smallest positive value is about 6e-8.
• bfloat16 keeps the value, roughly, because it has float32's range.
• Loss scaling by 1024 keeps the value in float16 too, which is the fix the mixed precision paper used.
• bfloat16 rounds 1 + 1/256 down to 1, so small updates to a weight vanish. That is one reason training keeps a 32-bit master copy of the weights.

Because XLA compiles a plan for each input shape, and compiling takes time. Google's Cloud TPU guide says recompiling every time the shape changes is too slow, and that dynamic shapes are not well suited to TPUs.

You can watch the compile count on a laptop, because JAX compiles through XLA on CPUs too. The test counts how often JAX traces, and so compiles, the function, which here happens once for each new input shape.

import jax
import jax.numpy as jnp

compiles = 0
W = jnp.ones((64, 32))

def embed(tokens):
    global compiles
    compiles += 1              # Python runs this only while JAX traces a new shape
    return jnp.tanh(tokens @ W).mean(axis=0)

lengths = [7, 9, 12, 7, 31, 9, 18, 5]

raw = jax.jit(embed)
for n in lengths:
    raw(jnp.ones((n, 64)))
print("raw lengths:", compiles, "compiles")

compiles = 0
padded = jax.jit(embed)
for n in lengths:
    size = 16 if n <= 16 else 32
    padded(jnp.pad(jnp.ones((n, 64)), ((0, size - n), (0, 0))))
print("two buckets:", compiles, "compiles")

On JAX 0.12.0 (CPU build, as of 19 October 2026), it prints 6 compiles for the raw lengths and 2 for the padded ones.

What just happened? Eight requests with six different lengths caused six compiles. Padding to two bucket sizes, 16 and 32 rows, cut that to two. The price: 62 of the 160 rows sent were padding, which is wasted maths.

Padding rows also change results such as this average. Real code passes a mask, a list of which rows are real, and leaves the padding out.

Google's guide describes a second kind of padding. When a matrix cannot fill an MXU, the compiler pads it with zeros, so it asks for a batch or feature size that is a multiple of 128.

Run your own workload on each option and divide the hourly rent by the useful results per hour. Peak speeds on spec sheets describe ideal conditions that real jobs rarely reach.

Benchmarks help, but read them with care. The MLPerf Inference paper says over 100 organisations were building inference chips. It says the systems span at least three orders of magnitude in power and five in performance.

That spread is why the paper argues for shared benchmark rules. Even the 2017 TPU paper reports low utilisation for some of its own applications. A chip with a huge peak can still sit mostly idle.

To compare options for the re-embed, the team writes a small cost check. It uses three made-up rents and speeds.

OPTIONS = {          # rent in Rs per hour, embeddings per second (made up)
    "cpu_server": (60, 400),
    "gpu_server": (320, 9000),
    "tpu_slice": (280, 11000),
}

def cost_per_crore(rent_per_hour, per_second, utilisation):
    useful_per_hour = per_second * 3600 * utilisation
    return rent_per_hour * 1e7 / useful_per_hour   # 1 crore = 1e7

for name, (rent, speed) in OPTIONS.items():
    print(name, round(cost_per_crore(rent, speed, 0.8)))

The key line is useful_per_hour. It multiplies speed by utilisation, because idle time is paid for but produces nothing. At 80 percent busy it prints about Rs 521 per crore on CPU, Rs 123 on GPU and Rs 88 on TPU.

Now set the GPU's utilisation to 0.05, like the live query servers. The GPU cost jumps to about Rs 1,975 per crore. A busy CPU beats an idle GPU.

So the team agrees on a short process before moving anything.

1. 1

   Split the bill by workload

   Tag every server as training, batch or live. The Rs 38 lakh turns out to be 45 percent training, 20 percent batch and 35 percent live.

2. 2

   Pick one metric per workload

   Training steps per hour for training, embeddings per hour for batch, and queries per second within 50 milliseconds for live.

3. 3

   Run a one-hour trial on each option

   Use real data, real lengths and the real model, with batching tuned for each chip.

4. 4

   Divide rent by useful output

   Include expected utilisation, so night-time idle hours are counted honestly.

5. 5

   Add the hidden costs

   Engineering time to port the code, data transfer between clouds, and the effort of running two kinds of hardware.

6. 6

   Decide, then recheck each quarter

   Prices, models and traffic change. A choice that was right in March can be wrong by October.

Here is where the team lands. Notice that each workload gets its own chip, and both embedding paths write to or read from one vector index.

The re-embed stays on GPUs, even though the TPU looked cheaper per crore. The backfill code uses a custom pooling step, the part that merges token vectors into one, and porting it would take three weeks.

At one backfill a quarter, the saving does not repay that time yet. The new monthly bill comes to about Rs 24 lakh, and training finishes in two days instead of four (both made up).

Finance asks for one thing in return: a usage graph per workload, so idle chips show up early next time.

Explain the choice per workload, tied to one number: how busy the chip stays doing useful maths. Then show you know the trade-offs beyond speed.

1. 1

   List the workloads

   Separate training, batch inference and live inference. Give each its size, its traffic pattern and its latency target.

2. 2

   Explain the core difference

   CPUs handle varied step-by-step work. GPUs run the same maths on many numbers at once. TPUs are built around fixed grids for matrix maths.

3. 3

   Talk about batching and memory

   Say that big chips need large batches to stay busy, and that memory bandwidth often limits speed more than raw maths.

4. 4

   Match chips to jobs

   Big steady jobs go to GPUs or TPUs. Small models with spiky live traffic can go to CPUs.

5. 5

   Name the non-speed costs

   Software support, porting effort and lock-in to one cloud all count.

6. 6

   Finish with measurement

   Propose a trial on real data and a cost per useful result, then a regular review.