Autoscaling Explained: Scale on the Right Signal

When autoscaling watches the wrong number, it adds copies too slowly or not at all while users wait. The autoscaler itself works perfectly. It is simply answering the wrong question.

Picture a common setup. A flash-sale shopping app drops a small batch of discounted kurtas, sarees and sneakers at 12 noon every day. Most drops sell out within ten minutes. The setup is made up, but the problem is real.

At 11:55 the checkout service gets about 110 requests a second. Thirty seconds after noon it gets about 3,200. It runs as 8 copies on Kubernetes, a system that runs containers across many machines.

An autoscaler is told to keep the checkout copies near 60 percent CPU use. Paid orders go into a queue. Order workers pick them up and call the payment gateway, the partner service that confirms each payment. The workers scale on CPU too.

On the day of the Diwali drop, three things went wrong in the first five minutes.

• The checkout service scaled, but slowly. Each copy was capped at the CPU it had reserved, so CPU use stopped at 100 percent. The autoscaler saw busy, never how busy.
• New copies took about 90 seconds to start and load the product catalogue. By the time they were ready, many shoppers had seen errors and left.
• The order workers never scaled at all. They spent most of their time waiting on the payment gateway, so their CPU stayed near 20 percent while 40,000 orders piled up.

The incident review was blunt. The autoscaler did exactly what it was told, and it was told to watch the wrong thing. The rest of this guide fixes each of these failures in turn.

Autoscaling is a system that changes how many copies of your service run, on its own, by watching a measured number and comparing it with a target. More load brings more copies, and less load removes them.

A few words first. A replica is one running copy of your service. In Kubernetes a replica is a pod: one or more containers that run together. On a cloud provider it is often a virtual machine, called an instance.

Scaling out means adding replicas, and scaling in means removing them. Together this is horizontal scaling. Vertical scaling instead gives one replica more CPU or memory. This guide is about horizontal scaling, the kind most autoscalers do.

The number the autoscaler watches is a metric: a value measured over time, such as CPU use or requests per second. The target is the value you want that metric to stay near.

Picture a bhutta seller on Juhu beach in Mumbai at sunset, roasting corn over small coal stoves. Each stove is a replica. The seller is the autoscaler. Whatever he looks at before lighting another stove is the metric.

As the evening crowd grows, he lights a second stove, then a third. When the crowd thins, he lets one burn out. A fresh stove needs several minutes before its coals are hot enough, which is his start-up time.

Where the picture breaks: the seller sees his whole crowd at a glance. A real autoscaler sees only the numbers you send it, sampled every few seconds. It never sees a crowd it was not told to count.

Autoscale on a metric that grows with the work waiting for you, and that more replicas will actually bring down. For many services that is requests per replica or queue backlog, not CPU.

Back on the beach, a stove roasting at full speed looks the same whether three people wait or thirty. Counting cobs roasted tells the seller he is busy. Counting people at his cart tells him how many stoves he needs.

CPU is the cobs-roasted number. In Kubernetes each container can reserve CPU, called its request, and can be capped by a CPU limit. When the limit equals the request, CPU use measured against the request cannot pass 100 percent.

The kernel, the core of the operating system, enforces that cap by slowing the container down, so a spike shows up as busy and nothing more. CPU also stays low while work waits on something else, like the workers waiting on the payment gateway.

The table compares the common choices, with the part of the flash-sale app where each one fits.

Memory deserves its own warning. In our test on Python 3.13, a process grew from 9 MiB to 232 MiB under load. A MiB, or mebibyte, is 1,048,576 bytes. After it dropped 99 percent of its objects, it still held 224 MiB.

Freeing everything did give the memory back. Keeping a few objects from the busy spell did not. A memory-based autoscaler would see that replica as nearly full long after the rush.

Response time needs a warning too. It feels like the most honest signal, because it is what users feel. But if the slowness comes from an overloaded database, adding replicas sends even more queries to that database.

The team measured what one checkout replica could handle with a load test, a test that sends fake traffic at rising rates. One replica coped with about 70 requests per second before responses slowed. They set the target at 50, leaving room for bursts.

The Kubernetes Horizontal Pod Autoscaler, or HPA, uses proportional maths: if the metric is double the target, it doubles the replicas. It multiplies the current replicas by the metric divided by the target, then rounds up.

The Kubernetes documentation writes it as desired replicas = ceil(current replicas × current value ÷ target value). Ceil means round up to the next whole number, because you cannot run half a pod. The current value is the average across the pods.

Take the flash sale with the new metric. There are 8 checkout pods, each getting 400 requests per second against a target of 50. So the HPA asks for 8 × 400 ÷ 50 = 64 pods.

Now compare the old CPU setup. CPU sat at 100 percent of the request against a target of 60. That gives 8 × 100 ÷ 60 = 13.3, rounded up to 14. The HPA asked for 14 pods when the sale needed 64.

Then it waited, measured again and asked for a few more. The engineer section near the end shows exactly how slow that climb was.

Here is the loop the HPA runs. Follow the arrows from the metric round to the change and back again. Notice the box where the autoscaler does nothing, because the numbers are already close enough.

Two numbers in that loop come from the Kubernetes documentation for v1.37. The loop runs every 15 seconds by default. It skips any change when the ratio of metric to target is within 0.1 of 1.0, which stops it fiddling over tiny wobbles.

You can check the tolerance with our runs of the v1.37.1 controller code. At 54 requests per pod the ratio is 1.08, so 8 pods stay 8. At 56 requests the ratio is 1.12, so the HPA asks for 9.

On a live cluster, kubectl, the Kubernetes command-line tool, shows the same numbers. Running kubectl get hpa checkout-api at the start of the spike shows TARGETS as 400/50 and REPLICAS as 8.

Scale queue workers on backlog per worker: the number of messages waiting, divided by the number of workers. Set the target from how long a message may wait and how long one takes to process.

A queue is a list of jobs waiting to be done, held by a separate service so the sender does not have to wait. The backlog is how many jobs are waiting right now. In the flash-sale app, each job is one paid order to confirm.

The AWS documentation on scaling with an Amazon SQS queue makes the key point. It warns that a policy targeting only the number of messages ignores how many instances you already have. That can add more instances than needed while the queue drains.

Its recipe is short. Divide the longest acceptable wait by the average time to process one message. The answer is the acceptable backlog per instance, and that becomes your target.

The team applies it. An order takes a worker 0.5 seconds, mostly spent waiting on the payment gateway. No order should wait more than 30 seconds. So one worker can carry a backlog of 30 ÷ 0.5 = 60 orders.

Here is that calculation as a small Python function. It is the same proportional idea as the HPA, written out so you can see each piece.

import math

def desired_workers(backlog, seconds_per_order, max_wait_seconds,
                    min_workers=2, max_workers=60):
    # Orders one worker can clear within the allowed wait
    per_worker = max_wait_seconds / seconds_per_order
    needed = math.ceil(backlog / per_worker)
    return max(min_workers, min(needed, max_workers))

# Noon: 3,000 orders waiting, 0.5 s each, 30 s allowed wait
print(desired_workers(3000, 0.5, 30))   # 50
# Quiet hour: 40 orders waiting
print(desired_workers(40, 0.5, 30))     # 2
# Huge spike: 9,000 orders waiting
print(desired_workers(9000, 0.5, 30))   # 60

Run it and it prints 50, 2 and 60, one per line. The per_worker line is the acceptable backlog per worker. The return line clamps the answer between a floor and a ceiling, so a quiet hour keeps 2 workers and a huge spike stops at 60.

In Kubernetes you rarely write this loop yourself. KEDA, an open-source event-driven autoscaler, reads queue length from systems such as Kafka, RabbitMQ and Amazon SQS. It feeds those numbers to the HPA.

KEDA wakes a workload from zero to one replica itself, and the HPA handles one to many. As of 19 October 2026 the latest release is KEDA 2.21, where minReplicaCount defaults to 0.

Its cooldownPeriod defaults to 300 seconds, and it applies only to the final step down to zero. That suits workers that sit idle most of the day.

Autoscaling is slow for spikes because several delays stack up. The metric arrives late, the loop runs on a timer, new machines take time to start, and new replicas must pass a readiness check first.

A readiness check is a test a pod must pass before it gets traffic. In Kubernetes, a pod that fails its readiness probe is taken out of its Service, the stable address that spreads requests across pods. A pod still loading its catalogue gets no shoppers.

Here is one noon spike before the fix. Read the boxes in order and add up the time in each note. The shopper is waiting through all of it.

Add it up and a new pod becomes useful about two and a half minutes after the drop. Most drops sell out in ten. So a quarter of the sale happens on too few pods.

Scale before the spike when you can see it coming

The noon spike is not a surprise, because it happens at 12:00 every day. Scheduled scaling raises capacity at a set time, before the load arrives, instead of waiting for a metric to climb.

On Kubernetes, KEDA's cron trigger sets a replica count between a start and an end time, in a time zone you name, such as Asia/Kolkata. When other triggers are present, that count acts as a moving minimum.

The team uses it to hold the checkout floor at 50 pods from 11:50 to 12:30. KEDA builds the HPA from its own settings, so the request-rate rule becomes a second trigger beside the cron one.

That second trigger keeps running, so a bigger drop than usual still gets more pods on top of the 50.

On virtual machines, AWS Auto Scaling groups offer scheduled actions that set capacity at a date and time. Repeats are written as cron expressions, a five-field code such as 50 11 * * * for 11:50 every day.

Watch the clock. AWS recurring schedules use UTC, Coordinated Universal Time, by default. So 11:50 in India is 06:20 UTC unless you set the time zone on the action.

Make new replicas start faster

When spikes are not predictable, shorten start-up instead. The team cut pod start-up from 90 seconds to 25 by loading the noon drop's products first and the full catalogue later.

For virtual machines, AWS offers warm pools: instances set up in advance and kept beside the group. Its documentation says that for stopped warm instances, you pay only for their disk volumes and attached Elastic IP addresses, the fixed public addresses AWS rents out.

The same page warns that a warm pool you do not need only adds cost. Use one when a slow first boot actually hurts users. Here are the main ways to beat the lag:

• Keep headroom. A lower target, such as 50 instead of 70 requests per pod, leaves spare room for the first minute of a spike.
• Raise the minimum on a schedule for spikes you can predict.
• Prepare machines in advance, with a warm pool or spare capacity, when boot time is long.
• Put a waiting room in front of a big sale: a page that lets shoppers in at a steady rate.

You stop flapping by scaling down slowly: wait until the metric has stayed low for several minutes before removing replicas. Flapping means the replica count bouncing up and down because the metric wobbles.

Flapping hurts in two ways. Every new replica pays the start-up cost again. And removing replicas just before the next burst means you start that burst short-handed.

Kubernetes handles this with a stabilization window for scaling down. The HPA remembers its recommendations over the window and uses the highest one. The default window is 300 seconds, or 5 minutes.

The flash sale has a special reason to wait longer. At 12:05, unpaid items in carts go back on sale, and a second wave arrives. So the team sets a 10-minute window and slow removal.

Here is the part of the checkout HPA that sets the metric and the scale-down behaviour. A metrics adapter supplies the requests-per-second number. It is a small service that exposes the app's own numbers to Kubernetes.

With KEDA, the same behavior block sits under advanced.horizontalPodAutoscalerConfig, and KEDA copies it into the HPA it creates. The fields and their meaning do not change.

# Part of the HPA spec for checkout-api
minReplicas: 8
maxReplicas: 80
metrics:
- type: Pods
  pods:
    metric:
      name: http_requests_per_second
    target:
      type: AverageValue
      averageValue: "50"
behavior:
  scaleDown:
    stabilizationWindowSeconds: 600
    policies:
    - type: Percent
      value: 10
      periodSeconds: 60

The metrics section targets an average of 50 requests per second per pod. The behavior section waits 600 seconds before scaling down, then removes at most 10 percent of pods per minute.

We ran these settings through the v1.37.1 controller code, with 64 pods and traffic falling to the 8-pod level. The first removal came after about 10 minutes, when 64 became 57. Then came 51, 45 and so on, reaching 8 after about 25 minutes.

With the defaults instead, the count held at 64 for about 5 minutes and then fell straight to 8 in one step.

The team leaves scale-up alone, because adding pods quickly is exactly what the sale needs.

Autoscaling cannot fix a bottleneck outside the replicas it controls, such as a full database, a rate-limited partner or a bug. It can make those worse, because more replicas send more load downstream.

In the second week, the checkout service scaled to 64 pods perfectly. Then the database fell over. Each pod opened 20 database connections, so 64 pods asked for 1,280. The database allowed only 500.

The team added a connection pooler, a service that shares a small set of database connections among many app replicas. They also checked that the pod maximum and the pooler settings could never go past the database's limit.

Other things to check before you trust an autoscaler:

• Set a maximum. Without one, a looping bug or a bot attack can grow your bill as fast as your traffic.
• Check partner limits. The payment gateway accepts only so many calls per second, so 200 workers would only collect more errors.
• Watch for a bad deploy. If a new version is slow, the autoscaler adds more slow copies, which hides the real problem.
• Check your cloud quota. AWS, for example, caps the vCPUs (virtual CPUs) of pay-as-you-go On-Demand instances each account can run per Region.
• Keep readiness checks honest. A replica that says it is ready before it can really serve will take traffic and fail.

A good setup gives each kind of work its own signal, adds known spikes on a schedule, and sets hard limits where downstream systems would break. Here is the flash-sale design after the fixes.

Follow the shopper from the top. Notice that each autoscaled service watches its own signal, and both paths end at limits the autoscalers must respect.

The team also built a dashboard with three lines: replica count, the scaling metric and its target. When the replica line lags behind the metric line, they know the autoscaler is slow rather than wrong.

On the next Diwali-sized drop, the service was at 50 pods before noon and reached 70 by 12:01. The workers grew to 45 as the backlog rose. The database stayed under its connection limit the whole time. These numbers are made up, like the setup.

It climbed slowly because each round could multiply the pods by only 100 ÷ 60, about 1.67, and the next round waited for the new pods to become ready. We ran the v1.37.1 replica calculator, the HPA's own code, on the sale's numbers.

It asks for nothing new. For CPU metrics, pods that are not yet ready are set aside, then counted as using 0 percent when the HPA wants to scale up. That pulls the average down to 57 percent.

Now the adjusted ratio says scale down, the opposite direction. The documentation says a ratio that reverses direction cancels the change, so the count stays at 14 until the new pods are ready.

If each new pod takes about 90 seconds to become ready, the climb to 67 ready pods takes at least 6 minutes. The drop sells out in ten.

The request-rate metric has no such ceiling. Its first decision asks for 64 at once. The default scale-up limit then allows 4 more pods or 100 percent more, whichever is higher, every 15 seconds.

In our run the requested count went 8, then 16, then 32, then 64 within 30 seconds. Each new pod still needs its start-up time before it serves. New pods with no metric yet count as 0 during scale-up, but the 8 busy pods still average 200 across 16, so the HPA still asks for 64.

One more detail shows on the screen. While pods are starting, kubectl get hpa keeps showing 400/50, because the HPA reports the raw average, not the adjusted one. Our run with 16 pods printed exactly that.

The readiness rule is narrower than it sounds. The documentation states it for CPU, and the v1.37.1 source agrees: only CPU sets aside running pods that are not ready. For other metrics, a running pod counts with whatever it reports.

Five settings matter most: the scale-up rate, the two stabilization windows, the scale-down rate and the tolerance. Each default suits a steady service, not a sale whose traffic jumps 30 times at noon.

The scale-up window is the one people misread. In our run, a 60-second window kept the count at 8 through a single high reading. It moved only once every reading in the past minute was higher.

The per-HPA tolerance field is stable from v1.37. With a scale-up tolerance of 0.05, our run asked for 9 pods at 54 requests per pod, where the default 0.1 kept 8.

Explain it as a loop with a carefully chosen signal, then show you know its delays and its limits.

1. 1

   Name the work and the signal

   For each service, say what work piles up and which metric tracks it: requests per replica for an API, backlog per worker for a queue.

2. 2

   Give the formula with numbers

   Desired replicas equal current replicas times current metric over target, rounded up. Work one example out loud, like 8 pods at 400 against a target of 50.

3. 3

   Set the minimum and maximum

   The minimum covers normal traffic and a failed replica or two. The maximum protects the database, partner limits and the bill.

4. 4

   Deal with the lag

   Add up the delays: metric, loop, machine, start-up and readiness. For known spikes, scale on a schedule. For unknown ones, keep headroom and start faster.

5. 5

   Stop the flapping

   Scale up quickly and down slowly, with a stabilization window of several minutes and a cap on how fast pods leave.

6. 6

   Say what it cannot fix

   Point at the shared database and the partners downstream. Explain the pooler and the caps that stop scaling from moving the failure somewhere else.