LLM Capacity Planning: Back-of-the-Envelope Maths

Replies slow to a crawl and then fail, because the GPUs running the model can only write so many words per second. Traffic jumps on one day, and extra GPUs or higher limits cannot be arranged in an afternoon.

Picture a common setup. An online test-prep app runs an AI doubt helper, where students type questions and get answers in chat. On a normal day about 40,000 students use it, and the helper runs on two rented GPUs. The setup is made up, but the problem is real.

Then the results date for a big entrance exam is announced for next month. The team expects 200,000 students in results week, asking about cut-offs, ranks and college choices. Last year a smaller spike pushed replies past a minute at 8 pm, and students gave up.

So the team needs two numbers before booking anything: how many GPUs, and what they will cost. This guide works both out on one page, with rough sums on purpose.

You need six words: token, requests per second, throughput, latency, GPU and headroom. Each one below has a plain meaning and an example from the test-prep helper.

• Token: a small piece of text the model reads or writes, often a word or part of a word. With three common tokenizers (tools that split text into tokens), a 194-word English answer came to about 220 tokens. The first few sentences of that answer, written in Hindi, took 2.7 to 7.4 times as many tokens as in English, so count with your own model's tokenizer.
• Requests per second (RPS): how many messages reach your system each second. 50 RPS means 50 student questions arrive every second.
• Throughput: how much work the whole system finishes per second, such as 2,000 tokens per second across all students together.
• Latency: how long one student waits for one reply. Throughput and latency often pull against each other, as the load test below shows.
• GPU (graphics processing unit): the chip that runs the model. The team rents GPUs with 80 GB of memory each, paid by the hour.
• Headroom: spare capacity you keep on purpose for bursts, failures and surprises, such as GPUs for 65 RPS when you expect 50.

LLM capacity planning means working out how much serving power a model-based feature needs at its busiest, and what that will cost. LLM stands for large language model, the kind of AI that reads and writes text.

It is a chain of estimates where each one feeds the next. Users become requests, requests become tokens, tokens become GPUs, and GPUs become rupees. If one number is wrong, every number after it is wrong too.

Back-of-the-envelope maths means rough sums you could do on a phone calculator. It tells you whether you need 2 GPUs, 9 or 90, and which number to measure first.

Multiply users by messages per user to get daily requests, take the busiest hour's share, and divide by 3,600 seconds. Then multiply that rate by the tokens each request carries.

The team starts with 200,000 students. Last year's logs show each active student sent about 6 messages to the helper on a busy day. That gives 1.2 million messages on the peak day (made up).

Spread evenly over 86,400 seconds, that is about 14 messages per second. That number is a trap, because nobody chats evenly through the night.

Why plan for the peak and not the average?

Think of the overhead water tank on a Jaipur rooftop. A thin supply line fills it slowly all day. At 7 in the morning, every tap in the house opens at once.

• The supply line filling the tank is the daily average: about 14 requests per second.
• Every tap open at 7 am is the busiest hour: 50 requests per second.
• The pipe from the tank to the taps is your serving capacity. It must match the taps, not the supply line.

Where the picture breaks: a tank stores water ahead of time, but a GPU cannot store answers before the questions arrive. Every reply is computed when it is asked for, so the capacity must exist at the peak itself.

Now the real numbers. The logs show the busiest evening hour carried about 15 percent of the day's messages. 15 percent of 1.2 million is 180,000, and 180,000 divided by 3,600 seconds is 50 requests per second.

The peak-to-average ratio is the peak rate divided by the average rate. Here it is 50 divided by 13.9, about 3.6. Your own ratio must come from your own logs, and event days tend to push it higher.

Peaks come from more places than the clock:

• Time of day: most students study from 7 pm to 10 pm.
• Events: results day, or a push notification sent to every user at once.
• Retries: when replies slow down, students press send again, which adds load at the worst moment.
• Bursts inside the hour: a single minute can be far busier than the hour's average.

How many tokens does each request carry?

Count input tokens (everything you send to the model) and output tokens (the reply it writes) separately. The team breaks one typical message into parts:

• System prompt, the fixed instructions sent with every message: 400 tokens.
• Recent chat history, so the helper remembers the conversation: 600 tokens.
• Cut-off notes fetched from the app's own data: 400 tokens.
• The student's question: 100 tokens.
• The reply: about 250 tokens, kept short by an instruction in the system prompt.

So each message carries about 1,500 tokens in and 250 out. At 50 requests per second, that is 75,000 input tokens and 12,500 output tokens every second.

It depends on the model, the GPU, the prompt sizes and how many chats run together, so you measure it with a load test. A load test sends test traffic at rising levels and records the speed.

A reply happens in two phases. First comes prefill: the model reads the whole input in one go and produces the first token. Then comes decode: it writes the rest of the reply one token at a time.

The model's learned numbers are called weights. The team's made-up model has 8 billion of them, stored at 2 bytes each, so about 16 GB. Every decode step must read all those weights from GPU memory.

Prefill keeps the GPU's maths units busy, because it handles all 1,500 input tokens at once. Decode makes only one token per chat per step. So a decode step is limited by how fast memory can be read, not by maths.

That sets a hard ceiling for a single chat. Memory bandwidth is how many bytes a GPU can read from its memory each second. The data sheet for the A100 80GB SXM, a module that sits on the server board rather than a plug-in card, lists 2,039 GB per second. Reading 16 GB at that rate takes about 8 milliseconds, so one chat alone tops out near 127 tokens per second on that card.

Two waiting times matter to a student. Time to first token (TTFT) is how long until the reply starts to appear. Time per output token (TPOT) is the gap between later tokens. Serving research, such as a 2024 paper on separating prefill from decode, sets its targets on exactly these two.

What does batching do to tokens per second?

Batching means the GPU works on many chats in the same step. One read of the weights now serves every chat in the batch, so total output rises a lot. But each step does more work and gets slower, so every student sees fewer tokens per second.

Pause and predict: if 64 chats share one GPU, does each reply take 64 times as long? Make a guess, then check the load test below.

The team runs a load test with real prompts, about 1,500 tokens in and 250 out. It raises the number of chats running together and records how long one reply takes and the total output.

What just happened? At 64 chats, each reply took about twice as long as a lone chat, not 64 times as long. That is batching at work: one read of the weights serves all 64. Past about 128 chats, total output stops rising while every student waits longer.

The team's rule is that each student should see at least 25 tokens per second on average. That is more than five times the average adult silent reading speed for English non-fiction, 238 words a minute or about 4.5 tokens per second. So it picks 64 chats per GPU, giving 2,000 output tokens per second.

Going to 128 chats would add about 15 percent more output and cut each student's speed by about 40 percent. That trade is rarely worth it for a chat feature.

A serving engine is the software that runs the model and handles incoming chats. Open-source engines such as vLLM let a finished chat leave after any step and a waiting chat join at the next one. This idea, iteration-level scheduling, was introduced in a 2022 paper. Our guide on LLM inference system design covers it in depth.

Divide the output tokens you need per second at peak by what one GPU gives at your chosen batch size. Then add headroom on top and round up.

At peak, the helper needs 12,500 output tokens per second. Each GPU gives 2,000 at 64 chats. 12,500 divided by 2,000 is 6.25 GPUs, if every GPU ran flat out all hour.

Running at 100 percent is a plan to fail. Bursts inside the hour, a crashed GPU and impatient retries all need room. The team adds 30 percent headroom: 6.25 times 1.3 is about 8.1, so it books 9 GPUs.

Check the failure case too. If one of the 9 GPUs dies at 8 pm, 8 remain, still above the 6.25 needed. That is what headroom buys.

Divide what you pay per hour by the chats you serve per hour, then multiply by 1,000. You pay for rented GPUs even when they sit idle, so how busy they are matters as much as the price.

Here a chat means one student question and its reply. The team's GPU costs Rs 250 per hour (made up). At full load one GPU writes 2,000 tokens per second, which is 8 replies of 250 tokens per second, or 28,800 an hour.

Rs 250 for 28,800 chats is about Rs 9 per 1,000, but only if the GPU is always full. Renting all 9 GPUs for the whole peak day costs Rs 54,000 for 1.2 million chats. That is Rs 45 per 1,000, because most hours are quiet.

The six hours from 4 pm to 10 pm carry most of the traffic, with the 7 pm to 10 pm study rush inside them. Outside them the logs show at most about 10 requests per second, which needs 2 GPUs by the same sum with headroom. The team keeps 3 all day so one can fail, and adds 6 for the evening.

That is 108 GPU-hours, or Rs 27,000, about Rs 22.5 per 1,000 chats. The extra 6 GPUs must be running and warm before the rush starts, not when it starts.

With a hosted model API, you pay per token instead of per GPU-hour. Say the service charges Rs 10 per million input tokens and Rs 40 per million output tokens (made up). One chat then costs Rs 0.015 for input plus Rs 0.01 for output, so Rs 25 per 1,000 chats.

Cutting these costs with caching, shorter prompts and smaller models is a subject of its own, covered in our guide on LLM cost optimisation.

Each chat needs memory for its KV cache, which grows with every token in the chat. GPU memory, minus the weights and the engine's working space, divided by memory per chat, gives the most chats one GPU can hold.

The KV cache is the model's working notes for one chat. For each token, every layer saves two sets of numbers, called keys and values. Keeping them means the model need not redo that work for every new token.

Inside each layer, the model splits its attention into parallel parts called heads. Many models let several heads share one set of keys and values, which shrinks the cache. This is called grouped-query attention, and the shared sets are the key-value heads.

Per token, multiply five numbers: 2 (keys and values), the layers, the key-value heads, the head size (numbers per head), and bytes per number. Bytes per number is 2 for 16-bit numbers, or 1 if you store the cache in 8-bit form. A model's config file lists them, often as num_hidden_layers, num_key_value_heads and head_dim. If head_dim is missing, vLLM 0.32.0 usually divides hidden_size by num_attention_heads, and so can you.

This matches how vLLM 0.32.0 sizes cache pages for standard attention layers. Models with sliding-window layers or compressed attention store less per token, so for them the formula overestimates.

The team's model has 32 layers, 8 key-value heads and a head size of 128 (made up). Its GPU has 80 GB. The team sets aside 8 GB for the engine's working memory and the share vLLM leaves unused.

def kv_cache_gb(tokens, layers=32, kv_heads=8, head_size=128, bytes_per_number=2):
    per_token = 2 * layers * kv_heads * head_size * bytes_per_number  # keys and values
    return tokens * per_token / 1e9

free_gb = 80 - 16 - 8              # GPU memory minus weights minus spare
per_chat_gb = kv_cache_gb(1_750)   # 1,500 tokens in plus 250 out
print(round(per_chat_gb, 2), int(free_gb // per_chat_gb))

The per_token line is the formula: 131,072 bytes, about 0.13 MB per token. A full 1,750-token chat needs about 0.23 GB. The script prints 0.23 244: the 56 GB left holds about 244 such chats.

So memory is not the limit at 64 chats per GPU, which need only about 15 GB. Speed per student is. That changes fast if prompts grow: at 8,000 tokens per chat, each needs about 1 GB and the same GPU holds only 53.

How do you check the estimate against the real engine?

vLLM prints two lines at startup that answer this directly. One reads "Available KV cache memory: N GiB". The other reads "GPU KV cache size: N tokens" followed by a maximum concurrency figure.

Divide the token count by your tokens per chat, 1,750 here, to get the chats that fit. If the log showed 427,000 tokens, that would be about 244 chats, the same as the sum above.

Two details trip people up. The memory line is in GiB (1,073,741,824 bytes), while the code uses decimal GB, so 56 GB shows as about 52 GiB. The concurrency figure assumes every chat reaches the longest length in tokens the server allows, so short chats fit far more.

By default, vLLM 0.32.0 lets itself use 92 percent of GPU memory, set by --gpu-memory-utilization (as of 19 October 2026). On an 80 GB card that leaves about 6.4 GiB unused, most of the team's 8 GB allowance.

Cards sold as 80 GB also hold more than 80 decimal GB. NVIDIA's Multi-Instance GPU user guide shows an H100 80GB card reporting 81,559 MiB, about 85.5 GB. So a real log shows a few GiB more KV room than this sum, which errs on the safe side.

Wasted memory used to make all this worse. The 2023 PagedAttention paper showed that serving systems wasted KV cache memory through fragmentation and duplicate copies. Storing the cache in small fixed-size pages, as an operating system does with memory, cut that waste to near zero.

Run a load test against your own server at several concurrency levels (how many chats run at once), with your real prompt and reply sizes. vLLM ships one, vllm bench serve, which sends random prompts of the lengths you choose.

vllm bench serve --model MODEL_NAME --dataset-name random \
  --random-input-len 1500 --random-output-len 250 \
  --max-concurrency 64 --num-prompts 640 \
  --percentile-metrics ttft,tpot,e2el

Repeat with --max-concurrency set to 1, 16, 64 and 128, and --num-prompts about ten times that number, to rebuild the load-test table. Ten times keeps the one-chat run near a minute instead of 45 minutes. The --percentile-metrics flag adds end-to-end latency, which the default report leaves out. From each run, read four lines of the report.

• Output token throughput (tok/s): the GPU's total output, the number your GPU count divides by.
• Mean TTFT (ms): how long a student waits for the first token, in milliseconds.
• Mean TPOT (ms): the gap between later tokens. 1,000 divided by it gives tokens per second while the reply streams; 30 ms means about 33. That is a little above the table's figure, which also counts the wait for the first token.
• Mean E2EL (ms): end-to-end latency, the time from sending a question to its last token. Divided by 1,000, it is the table's seconds per reply, and 250 divided by those seconds gives what each student sees.

With random prompts, the tool tells the server to ignore the end-of-text token, so every reply runs the full 250 tokens (vLLM 0.32.0 source, as of 19 October 2026). Random prompts also share no common start. Prefix caching, on by default, reuses the stored KV cache of a shared start such as the system prompt, so real traffic may prefill faster.

Little's law, a rule from queueing maths, says the average number of items inside a system equals the arrival rate times the average time each one stays. It holds for long-run averages in a steady system, not for every second.

At peak, 50 chats arrive per second and each stays about 8 seconds, from question to last token. That is the load-test time at 64 chats; lighter load is quicker, so it errs on the safe side. So about 400 chats are in flight at once. Spread over 9 GPUs, that is about 45 chats per GPU.

Now compare 45 with the two limits you already have. It is under the 64 chats where each student still gets 25 tokens per second or more. It is far under the 244 chats that memory allows.

The load test obeys the same law. 64 chats each staying 8 seconds means 8 replies finish per second, which is 2,000 tokens per second. So speed and chats in flight are one check seen two ways, and memory is the independent one.

Here is the whole plan as a small Python function, so you can change one guess and rerun it. It uses only Python's built-in math module.

import math

def plan_gpus(daily_users, msgs_per_user, peak_share, out_tokens,
              gpu_tokens_per_sec, secs_per_chat, headroom=0.3):
    peak_rps = daily_users * msgs_per_user * peak_share / 3600
    need = peak_rps * out_tokens / gpu_tokens_per_sec   # GPUs at 100% busy
    gpus = math.ceil(need * (1 + headroom))
    in_flight = peak_rps * secs_per_chat                # Little's law
    return round(peak_rps, 1), round(need, 2), gpus, math.ceil(in_flight / gpus)

print(plan_gpus(200_000, 6, 0.15, 250, 2_000, 8))

It prints (50.0, 6.25, 9, 45). That is 50 requests per second, 6.25 GPUs if they ran flat out, 9 GPUs with headroom, and 45 chats per GPU at peak. Compare that last number with your load test and memory limits.

A hosted API caps how much your account may send, usually in requests per minute and tokens per minute. If your peak is above that cap, the extra requests are refused until the limit is raised, even with your bill fully paid.

Requests per minute (RPM) counts calls. Tokens per minute (TPM) counts tokens, and services differ on which tokens they count. Each service publishes its own limits and rules, so read the page for your account.

The team converts its peak. 50 requests per second is 3,000 RPM. Each request carries 1,750 tokens, so the peak needs about 5.25 million TPM. If the account allowed 2 million TPM, more than half the peak traffic would be refused.

A refused request usually comes back with HTTP status 429, Too Many Requests. RFC 6585, the standard that defines it, says the reply may include a Retry-After header saying how long to wait. A good client waits that long instead of retrying at once.

Here is how the helper handles a refused request. Follow the No arrow: the request waits and tries again, but never more than three times.

Retries handle short bursts, not a peak that is twice your limit. For that, the team starts weeks early:

• Ask for a higher limit weeks ahead, and treat it as uncertain until it is granted in writing.
• Ask whether the service sells reserved capacity, a fixed amount bought in advance.
• Cut tokens per request. Trimming history from 600 to 300 tokens lowers TPM by about 17 percent.
• Keep a fallback, such as a second model service or a small self-hosted model, for the busiest hour.

It should slow down gently and say so, instead of crashing. A per-student limit trims bursts, a waiting line holds chats the GPUs cannot take yet, and dashboards show when to act.

The team writes down what the helper does, in order, if the waiting line keeps growing:

1. 1

   Show the wait

   When the expected wait passes 30 seconds, the app tells students the helper is busy and shows their place in line.

2. 2

   Shorten replies

   Ask for shorter answers in the system prompt, about 150 tokens instead of 250, and lower the token cap to match. Each chat finishes sooner, so more chats get through each minute.

3. 3

   Trim history

   Send only the last two turns of the chat. That cuts input tokens and memory per chat.

4. 4

   Send simple questions to a smaller model

   A smaller, faster model answers routine questions, such as how to check a result online.

Walk the chain out loud with round numbers, state each assumption, and finish with headroom and what happens when you are wrong.

1. 1

   Ask for the load

   Daily users, messages per user, and when the peak hits. If there are no numbers, propose round ones and say so.

2. 2

   Get peak requests per second

   Daily requests times the busiest hour's share, divided by 3,600. Name the peak-to-average ratio.

3. 3

   Count tokens in and out

   Break the prompt into parts, and say that output tokens set the pace.

4. 4

   Size the GPUs

   Divide peak output tokens per second by one GPU's measured output at your batch size, then add headroom.

5. 5

   Check memory and chats in flight

   Use Little's law for chats per GPU, and compare with the KV cache limit.

6. 6

   Price it and plan for overload

   Give cost per 1,000 chats at full and realistic use, compare a hosted API's rate limits, and describe the waiting line.