Self-Hosting LLMs: Serving, Batching and GPU Autoscaling

Two things go wrong as you grow: the bill rises with every token, and every client file leaves machines you control. Sooner or later someone asks whether to self-host LLM inference instead, on GPUs of your own.

Picture a common setup. A legal-drafting product serves lawyers at about 40 law firms. They upload case files, contracts and court orders, and it drafts legal notices, summarises judgments and suggests clauses. The setup is made up, but the problem is real.

Every request goes to a model service: a company that runs a large model on its machines and charges per token. The product sends about 150 million tokens a day. The bill is about Rs 9 lakh (900,000 rupees) a month.

Then two things happen in the same week. A large client firm asks for a written promise that its files never leave machines the team controls. And finance notices that the bill has doubled in six months.

So the team asks a plain question. Should it rent its own GPUs and run an open-weight model itself? This guide follows the maths, the serving setup and the autoscaling needed to answer it.

You need seven words. Each one gets a plain meaning here, and the guide uses the same word for the same thing throughout.

• Token: a small piece of text, often a word or part of one. Model services count both what you send (input) and what comes back (output).
• GPU (graphics processing unit): a chip that does thousands of small sums at the same time. Its own memory holds the model and the work in progress.
• Open-weight model: a model whose trained numbers, called weights, you can download and run on your own machines.
• Inference server: the program that loads the weights onto a GPU and turns requests into tokens. vLLM is one open-source example.
• Throughput: how many tokens per second the whole system produces, across all users together.
• Latency: how long one user waits. Time to first token is the wait before words start to appear.
• Utilisation: the share of time and capacity your GPUs spend on useful work. A GPU busy 9 hours out of 24 still bills for 24.

Self-host LLM inference when you have high, steady token volume, data that must stay under your control, or both. With low or spiky traffic, paying per token is usually cheaper and far simpler.

Think of a builder deciding about a JCB, the yellow digger seen on Indian building sites. Hiring one by the hour costs more per hour, but you pay nothing on days you do not dig. Owning one means a loan, an operator and servicing every month, whether it digs or sits in the yard.

Here is how the picture maps onto the decision:

• Hiring by the hour is paying per token: the cost follows use.
• Owning the digger is renting or owning GPU servers: the cost runs every hour.
• The operator is the engineer who keeps those servers running and answers the 2 a.m. alert.
• Keeping the digger on your own site is keeping client files on machines you control.

Where the picture breaks: a digger does the same work whoever owns it. Models differ, and the open-weight model you can run may draft worse than the service's model. So quality needs its own test.

These are good reasons to self-host:

• A contract or a law says the data must stay on machines you control, in a region you choose.
• Your token volume is high and steady through the day, so the GPUs stay busy.
• You need a fine-tuned model, meaning one trained further on your own examples, and no service will host it the way you need.
• You need predictable latency that does not depend on another company's busy hours.

These are weak reasons, and they often end in a bigger bill:

• You want to save money but have never measured your real token volume.
• Traffic is small, or it comes in short bursts with long quiet gaps.
• Nobody on the team can be on call for GPU servers at 2 a.m.
• The best open-weight model you can run is clearly worse on your own tasks.

Self-hosting costs a fixed amount per GPU hour plus the people who run it, while paying per token costs a rate times your volume. Which one is cheaper depends on your volume, your peak traffic and how busy you keep the GPUs.

The team starts with what it can measure. It sends 150 million tokens a day, and at the late-morning peak it needs about 4,500 tokens per second. The service charges about Rs 200 per million tokens, input and output averaged. All of these figures are made up.

On the self-hosted side, load tests show one GPU server handles about 2,000 tokens per second with good batching. Renting one costs Rs 250 an hour. An engineer's time and monitoring tools add about Rs 1.5 lakh a month.

Count tokens per second the same way on both sides. Here it means input and output tokens together, on the team's real mix of requests. A server reads input much faster than it writes output, so a load test with short replies flatters the GPU.

The script below compares the two monthly bills. It sizes the GPUs for the peak plus 30 percent headroom, which is spare room for surprises. It never runs fewer than two servers, so one can fail without an outage.

import math

def api_bill(tokens_per_day, rs_per_million, days=30):
    return tokens_per_day * days / 1_000_000 * rs_per_million

def gpus_needed(peak_tps, tps_per_gpu, headroom=1.3, min_gpus=2):
    return max(min_gpus, math.ceil(peak_tps * headroom / tps_per_gpu))

def self_host_bill(gpus, rs_per_gpu_hour, people_rs, hours=24 * 30):
    return gpus * rs_per_gpu_hour * hours + people_rs

gpus = gpus_needed(peak_tps=4_500, tps_per_gpu=2_000)
print("GPUs:", gpus)                                    # GPUs: 3
print("Pay per token:", api_bill(150_000_000, 200))     # Pay per token: 900000.0
print("Self-host:", self_host_bill(gpus, 250, 150_000)) # Self-host: 690000

Run it with python3 and you should see the three lines shown in the comments: 3 GPUs, 900000.0 rupees for the service and 690000 rupees for self-hosting.

What just happened? gpus_needed sized for the busiest second, not the average, because most lawyers work the same office hours. The daily average is only about 1,736 tokens per second, so the peak is 2.6 times higher.

The min_gpus=2 setting is the floor you pay for even at 3 a.m., when nobody is drafting. The result is Rs 9 lakh for the service and Rs 6.9 lakh for self-hosting: about Rs 2.1 lakh saved each month.

The service costs Rs 2.4 lakh a month. Self-hosting still needs the two-GPU floor and the engineer, so it costs Rs 5.1 lakh. That fixed floor is what sinks small setups.

You can turn the floor into a break-even line. Rs 5.1 lakh a month buys about 85 million tokens a day at Rs 200 per million. Below that volume, this two-server setup cannot win on cost alone.

Sizing for the peak is its own skill, usually called LLM capacity planning. Every number in the script should come from a month of real logs, not from this guide.

A self-hosted LLM serving setup has a gateway that checks callers, a queue, GPU servers running an inference server, and an autoscaler that watches the load. Each part does one job.

The team picks an open-source inference server such as vLLM, so it writes configuration, not model code. The server also does the batching, which the next section explains.

Follow a lawyer's request in the diagram below, from the drafting app to a GPU server.

The team puts a queue in front of the GPUs on purpose. When every server is full, a request waits a few seconds instead of failing. The number of waiting requests also becomes the main signal for autoscaling.

Batching groups many users' requests so the GPU works on all of them in the same step. A model writes its reply one token at a time, and each step reads the model's weights once, however many requests share it.

Here is why that matters. Reading the weights from GPU memory is the slow part of writing a token. The Splitwise paper found that, even with batching, this writing phase leaves the GPU's compute underused.

So extra requests in a step share one read of the weights and use that spare compute. Each request still reads its own cached notes, so the sharing is large but not free.

The naive server handles one request at a time. It is simple and correct, but it pays the full weight read for every token of every user. That is the wrong way to run a shared GPU.

Back to the JCB. One scoop costs about the same diesel whether the bucket is full or holds one shovel of earth. A good operator fills the bucket on every scoop, and a good inference server fills every batch.

Static batching versus continuous batching

Static batching collects a group of requests, runs it until every request finishes, then starts the next group. The trouble is reply length. A two-line answer finishes early, but its slot sits empty until a long contract draft is done.

Continuous batching fixes this by changing the batch at every step. A finished request leaves at once, and a waiting one takes its place on the next step. The Orca paper (OSDI 2022) called this iteration-level scheduling.

On a 175-billion-parameter model, Orca reported 36.9 times the throughput of NVIDIA FasterTransformer at the same latency. That figure is the paper's own test, not a promise for your traffic.

In the diagram, start at the new request and follow it until it leaves the batch.

Paying per token is better when your volume is low or spiky, when you need the strongest models, or when nobody can run GPUs. It turns a fixed monthly cost into a cost that follows usage.

The JCB rule still holds. A small builder who digs two days a month should hire. Many teams start on a model service, measure for a few months, then move only the steady or sensitive part onto their own GPUs.

The diagram below is the order the team uses to decide. Start at the top.

The team's final decision is a split. Drafting from client files, for the firms that asked, goes to the self-hosted model. General legal research questions with no client data stay on the model service, which answers them better.

Every request in a batch needs GPU memory for its KV cache, so memory usually fills before compute does. The KV cache stores numbers the model already worked out for earlier tokens, so it does not redo them at each step.

The cache grows with every token. The PagedAttention paper works it out for a 13-billion-parameter model stored in 16-bit numbers: about 800 KB per token. One 2,048-token request then needs about 1.6 GB of cache.

Long case files make this worse. At that same rate, 32,000 tokens of case file would need about 26 GB, more than half of a 40 GB GPU.

That 800 KB rate belongs to the paper's model, and it changes with the model's shape. Each layer has several heads, parallel parts that look back at earlier tokens, and some models let several heads share one stored key-value head. Per token, the cache holds 2 (a key and a value) x layers x key-value heads x numbers per head x bytes per number. A model with 32 layers and 8 key-value heads of 128 numbers each, at 2 bytes per number, needs about 131 KB per token: 4.2 GB for 32,000 tokens.

The same paper found that older serving systems used only 20.4 to 38.2 percent of their KV cache memory for real token data. The rest went to space reserved for future tokens and to fragmentation, meaning gaps too scattered to use.

Its fix, PagedAttention, stores the cache in small fixed-size blocks, like pages in an operating system. The authors built vLLM around it and reported 2 to 4 times the throughput of FasterTransformer and Orca at the same latency.

The limits you actually set

You do not build any of this; you set limits and test them. In vLLM 0.32.0 these four flags matter most:

• --gpu-memory-utilization: the share of GPU memory this vLLM instance may use for the model, its working memory and the KV cache. The default is 0.92 in 0.32.0.
• --max-model-len: the longest request in tokens, input plus output together.
• --max-num-seqs: the most requests in one step's batch.
• --max-num-batched-tokens: the most tokens processed in one step, which bounds how much new input fits beside the replies being written.

Then test the limits with your largest real case files, not with short prompts. A limit that holds for a two-page notice can fail for a 40-page judgment.

The two phases of every request

Each request runs in two phases. Prefill reads the whole input at once, which keeps the GPU's compute busy. Decode then writes the reply one token at a time, which mostly waits on memory.

The Splitwise paper (2023) measured this difference and proposed running the two phases on separate machines. For the team the lesson is smaller. A 40-page case file means a long prefill, which can delay the next tokens of everyone sharing that GPU.

vLLM 0.32.0 turns on chunked prefill by default for models that support it. A long input is then processed in pieces, limited by --max-num-batched-tokens, so other users' replies keep moving.

You autoscale GPUs for LLM inference by watching how many requests are waiting, adding servers before the queue grows, and removing them slowly. The GPU busy percentage is a poor signal on its own.

The busy percentage misleads because NVIDIA's NVML library defines GPU utilisation as the percent of time one or more kernels, the small programs a GPU runs, were executing. It says that work was running, not how much.

So a server running 3 requests and one running 30 can both show close to 100 percent. Waiting requests and time to first token show what users actually feel. vLLM exports both on its metrics page.

The Kubernetes rule, exactly

Kubernetes runs containers, meaning packaged apps, across many machines. Its Horizontal Pod Autoscaler (HPA) adds or removes copies of an app, called replicas, based on a metric you choose. Here each replica is one GPU server.

The documented rule is a ratio. Desired replicas equal current replicas times the current metric divided by the target, rounded up. The controller checks every 15 seconds by default.

It skips any change when that ratio is within 10 percent of 1.0, the default tolerance. In Kubernetes 1.37 a per-direction tolerance field is stable, so scale-up and scale-down can differ. Scale-up has no waiting window by default.

Scale-down uses a 5-minute stabilization window (the Kubernetes spelling) by default and picks the highest recommendation seen in it. That window stops it from removing a server during a short lull, only to need it back a minute later.

Two practical details matter. A per-server metric such as waiting requests reaches the HPA through the custom metrics API, which a metrics adapter you install provides. And GPUs go in the limits section of a container's config, here one GPU per replica.

The team copies the rule into a small script, so it can test settings on last month's traffic before touching the cluster. The metric is waiting requests per server, averaged across servers, and the target is 8.

import math
from collections import deque

TOLERANCE = 0.1             # Kubernetes default: ignore changes within 10%
recent = deque(maxlen=20)   # 20 checks x 15 seconds = 5 minutes

def desired_replicas(current, waiting_per_server, target=8, low=2, high=6):
    ratio = waiting_per_server / target
    if abs(ratio - 1) <= TOLERANCE:
        want = current                      # close enough to target
    else:
        want = math.ceil(current * ratio)    # the Kubernetes ratio rule
    want = max(low, min(high, want))
    recent.append(want)
    if want < current:
        return min(current, max(recent))    # shrink only to the 5-minute high
    return want

print(desired_replicas(current=3, waiting_per_server=13))   # morning rush: 5
print(desired_replicas(current=5, waiting_per_server=1))    # short lull: stays 5
print(desired_replicas(current=5, waiting_per_server=8.5))  # near target: stays 5

What just happened? In the rush, 3 servers averaging 13 waiting gives 3 x 13 / 8 = 4.875, rounded up to 5. In the lull the ratio asks for 1 and the floor raises it to 2, but the 5-minute high is still 5.

Near target, 8.5 is within 10 percent of 8, so nothing changes. All three calls print 5.

The script leaves out two things the real HPA does. By default it adds at most 4 servers or 100 percent of the current count per 15 seconds, whichever is larger. It also handles servers that are not yet ready, or are missing metrics, more carefully than this script.

How do you check the signal on a running server?

Ask one vLLM server for its counts. vLLM serves its metrics page on the same port as its API, 8000 by default. Run this on the server:

curl -s localhost:8000/metrics | grep -E "^vllm:num_requests_(waiting|running)\{"
# vllm:num_requests_running{engine="0",model_name="MODEL"} 12.0
# vllm:num_requests_waiting{engine="0",model_name="MODEL"} 3.0

You should see two lines like the comments, with your model's name in place of MODEL. Running is how many requests are in the batch now; waiting is how many are queued inside that server.

If waiting stays above zero while running is far below --max-num-seqs, the limit is usually KV cache space or the per-step token budget. To confirm, check vllm:num_requests_waiting_by_reason: reason="capacity" counts those requests, and reason="deferred" counts ones held back for other short-lived reasons. Then check vllm:kv_cache_usage_perc, where 1.0 means 100 percent full.

Cold starts: the slow part of scaling up

A new GPU server is not ready in seconds. It must start, fetch the weights, load them into GPU memory and warm up. For the team this takes about 6 minutes (made up).

For large models, reading the weights is a big part of that. At 2 bytes per number, an 8-billion-parameter model is 16 GB and a 70-billion-parameter model is 140 GB. At an assumed 1 GB per second, the larger one needs 140 seconds just to read.

So the team reacts to early signals. It keeps the waiting target low and keeps the weights on fast storage near the servers. A scheduled job also raises the autoscaler's minimum to 3 servers at 9:30 every weekday morning.

The diagram shows one round of the team's autoscaler. Follow both question boxes: one decides when to grow, and the other decides when it is safe to shrink.

With the morning schedule and the autoscaler, the team runs about 3 servers for 12 office hours and 2 for the other 12. The GPU bill drops from Rs 5.4 lakh to Rs 4.5 lakh a month, so the total becomes about Rs 6 lakh. Short rushes above 3 servers add Rs 250 per extra server-hour on top.

Self-hosting moves every failure from the service to your team: GPU faults, memory errors, slow cold starts, model upgrades and security patches. Each one needs a guard that you build and test.

• Out of GPU memory. A few huge case files fill the KV cache and requests stall. Guard: cap --max-model-len and batch size, and test with your largest real files.
• One server dies. A third of your capacity vanishes at 11 a.m. Guard: never run fewer than two servers, ideally in different data centres.
• A rush beats the autoscaler. Fifty lawyers log in together after a big court ruling. Guard: warm headroom, a queue and a clear message that the draft is coming.
• An unbounded queue. In vLLM 0.32.0 the server's own queue has no limit unless you set --max-num-queued-reqs, which counts running and waiting requests together. Guard: set it to roughly --max-num-seqs plus the queue depth you accept, multiplying --max-num-seqs by the number of model copies if one server runs several side by side. Extra requests then get an HTTP 503 (busy) reply that a load balancer or client can retry elsewhere.
• No GPUs to rent. In some regions, rented GPUs can run short at busy times. Guard: reserve your floor capacity in advance.
• Quality changes after an upgrade. A new model version drafts clauses differently. Guard: rerun the 300 graded drafting tasks before every model change.
• Quiet cost creep. Test servers left running cost money every hour. Guard: a daily report of GPU hours against tokens served.

Explain it as a trade-off between cost and control first, then show the serving design that makes the numbers work.

1. 1

   Ask about volume and data

   Ask for tokens per day, the busiest hour, and whether data may leave the company. These three answers decide most of the design.

2. 2

   Do the break-even maths

   Compare a per-token bill with GPU hours plus people. Size GPUs for the peak with headroom, and say the two-server floor out loud.

3. 3

   Draw the serving path

   Draw the gateway, queue, load balancer, GPU servers running an inference server, and a store for the model weights.

4. 4

   Explain batching and memory

   Say that continuous batching keeps the GPU full, and that KV cache memory caps how many requests fit in a batch.

5. 5

   Add autoscaling

   Scale on waiting requests and time to first token. Scale up early, scale down after a stabilization window, and mention cold starts.

6. 6

   Name the failures

   Cover memory errors, a lost server, a rush faster than scale-up, a bounded queue, and quality checks before every model upgrade.