Autoscaling

Traffic is never flat. An online shop at 3 a.m. needs a fraction of the capacity it needs during a sale, and a batch queue can go from empty to a million messages in a minute. Autoscaling adds capacity when demand rises and removes it when demand falls, so you neither drop requests nor pay for idle machines. It is also a control loop that reacts to signals, and anything that reacts to signals can be pushed around by whoever controls them. In this lesson you'll learn how the Kubernetes Horizontal Pod Autoscaler computes replica counts (with worked numbers), how VPA, cluster autoscaling and KEDA fit around it, how to pick a scaling signal, and how to set limits so autoscaling can't become a denial-of-service or a denial-of-wallet.

#Vertical versus horizontal

There are two ways to add capacity:

Horizontal scaling is the default for services you write, and it builds directly on the previous lesson: each new replica becomes another backend behind the load balancer. Underneath both, the nodes themselves must scale too, or new Pods have nowhere to run. That's the job of cluster autoscaling.

#The Horizontal Pod Autoscaler

The HPA is a controller in the Kubernetes control plane. Every 15 seconds (the default sync period) it reads a metric for the Pods of a target workload, computes how many replicas would bring that metric to the target, and updates the workload's replicas field. Resource metrics (CPU and memory) come from metrics-server, which scrapes every kubelet; custom metrics such as requests per second come from an adapter (for example Prometheus Adapter or KEDA) that implements the custom or external metrics API.

Here is the whole loop, including what happens when the cluster runs out of room. Notice that the HPA never creates Pods or nodes itself: it only changes one number, and other controllers react to it.

#The formula, worked through

The HPA's core calculation is:

desiredReplicas = ceil( currentReplicas × currentMetricValue / targetMetricValue )

Example 1, scale up. 4 replicas, average CPU utilisation 90% of requests, target 60%. ceil(4 × 90 / 60) = ceil(6.0) = 6. Six replicas sharing the same total work would each run at 60%.

Example 2, scale down. Later, 6 replicas average 25%. ceil(6 × 25 / 60) = ceil(2.5) = 3. Rounding up always errs toward more capacity.

Example 3, tolerance. 5 replicas at 64% against a target of 60%: the ratio is 64 / 60 ≈ 1.07. The HPA ignores ratios within 10% of 1.0 (between 0.9 and 1.1), so nothing happens. Without this tolerance, normal noise would cause constant small changes.

Example 4, a per-Pod custom metric. Target 50 requests per second per Pod. 4 Pods are serving 400 RPS in total, an average of 100. ceil(4 × 100 / 50) = 8.

Example 5, several metrics. If an HPA has both a CPU target (which recommends 6) and an RPS target (which recommends 8), it computes each and takes the largest, 8. Then it clamps to minReplicas/maxReplicas.

The HPA is deliberately cautious with incomplete data. When scaling up, Pods that are not yet Ready or have no metrics are counted as using 0% of the target; when scaling down, Pods missing metrics are counted as using 100%. It also ignores CPU samples from Pods that have only just started, so a start-up CPU spike doesn't trigger a scale-up storm, which only works if your readiness probe is honest.

#An autoscaling/v2 HPA

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api
  namespace: shop
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "50"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
        - type: Pods
          value: 4
          periodSeconds: 60
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 20
          periodSeconds: 60

minReplicas: 3 keeps three replicas even at night, for availability; add a topology spread constraint on topology.kubernetes.io/zone if they must land in different zones. The Pods metric needs a custom metrics adapter; without one, delete that block.

The behavior section controls how fast the HPA moves. A stabilisation window makes the HPA look back over its recent recommendations: for scale-down with a 300-second window, it uses the highest recommendation from the last five minutes, so a brief lull doesn't remove capacity you'll need again in a minute. Policies cap the rate of change: here scale-up may double the replica count or add 4 Pods per minute, whichever is more (selectPolicy: Max), and scale-down may remove at most 20% per minute. The defaults, if you omit behavior, are no scale-up stabilisation (up to +100% or +4 Pods every 15 seconds) and a 300-second scale-down window.

The simulator below runs this loop over 30 minutes of traffic with the default behavior, so you can check its decisions against the formula. Watch the gap between the dashed desired line and the solid Ready line: that's Pod start-up time, and requests drop while the HPA is still catching up.

Check what the HPA is doing with kubectl:

$ kubectl -n shop get hpa orders-api
NAME         REFERENCE               TARGETS                                         MINPODS   MAXPODS   REPLICAS   AGE
orders-api   Deployment/orders-api   cpu: 90%/60%, http_requests_per_second: 61/50   3         30        4          3h

$ kubectl -n shop describe hpa orders-api | tail -3
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  12s   horizontal-pod-autoscaler  New size: 6; reason: cpu resource utilization (percentage of request) above target

#Why requests must be set

averageUtilization: 60 means "60% of the container's CPU request". If the containers in the Pod have no CPU request, there is nothing to divide by: the HPA can't compute utilisation and does nothing. You'll see cpu: <unknown>/60% and a FailedGetResourceMetric event saying the request is missing. Requests (introduced in Kubernetes fundamentals) are therefore not optional for CPU-based autoscaling. They also need to be realistic: a request far below real usage makes every Pod look like it's at 300% and the HPA will race to maxReplicas. If a sidecar skews the Pod's average, use type: ContainerResource to scale on one container's usage.

#Vertical and cluster autoscaling

The Vertical Pod Autoscaler watches actual usage and recommends (or applies) better requests. It's an add-on, not part of core Kubernetes:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: orders-api
  namespace: shop
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: "*"
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: "2"
          memory: 2Gi

updateMode: "Off" only publishes recommendations (kubectl describe vpa orders-api), which is the safest way to right-size requests. Initial applies them when Pods are created, and Recreate evicts Pods to apply new values; newer VPA releases add InPlaceOrRecreate, which uses in-place Pod resizing to avoid restarts. Don't let VPA and HPA both act on CPU or memory for the same workload: VPA raises the request, utilisation drops, the HPA scales in, and they fight.

Cluster autoscaling adds and removes nodes. Two tools dominate:

Both trigger on Pending Pods and both respect PodDisruptionBudgets when draining nodes. Karpenter's NodePool also carries a hard ceiling, which you'll want for cost control:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      expireAfter: 720h
  limits:
    cpu: "200"
    memory: 800Gi
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m

Outside Kubernetes, the same idea is a cloud autoscaling group: AWS Auto Scaling groups, Google Cloud managed instance groups and Azure VM scale sets keep a fleet between a minimum and maximum and adjust it with policies such as "target tracking: keep average CPU at 50%". The design questions (signal, warm-up, limits) are identical.

#Event-driven scaling with KEDA

CPU is a poor signal for queue workers: a worker blocked on I/O uses little CPU while 50,000 messages pile up. KEDA (Kubernetes Event-Driven Autoscaling) scales on the source of the work: queue length, stream lag, a Prometheus query, a cron schedule. It feeds those values to an HPA it manages, and it handles scaling between zero and one replica itself, which the HPA can't do.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: invoice-worker
  namespace: shop
spec:
  scaleTargetRef:
    name: invoice-worker
  minReplicaCount: 0
  maxReplicaCount: 50
  pollingInterval: 15
  cooldownPeriod: 300
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: keda-aws
      metadata:
        queueURL: https://sqs.eu-west-1.amazonaws.com/111122223333/invoices
        queueLength: "20"
        awsRegion: eu-west-1
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-aws
  namespace: shop
spec:
  podIdentity:
    provider: aws

queueLength: "20" is the target number of messages per replica. With 300 messages waiting, KEDA asks for ceil(300 / 20) = 15 workers; with 1,000 it would ask for 50; with 4,000 it still gets 50, because maxReplicaCount wins. When the queue has been empty for cooldownPeriod seconds, it scales to zero. Authentication uses workload identity (IRSA or EKS Pod Identity) rather than stored keys, as you learned in cloud IAM, and the role only needs sqs:GetQueueAttributes on that one queue.

#Choosing a signal, and living with start-up time

The signal should rise before users suffer and fall when the load falls.

Scaling on latency deserves extra suspicion. If p99 latency rises because the database is saturated, adding Pods adds database connections and makes it worse: a positive feedback loop.

Warm-up matters. A new replica isn't useful until the image is pulled, the process has started, caches are warm and the readiness probe passes. A JVM service might take 40 seconds; a new node adds one to three minutes more. Your target must leave headroom for that delay: a 60% CPU target means existing Pods can absorb a 60% surge while new ones start. Keep images small, use a startup probe instead of a long liveness delay, and for predictable peaks scale ahead of time (KEDA's cron trigger or a scheduled minReplicas increase).

Scale to zero saves money for spiky or rare work, but the first request after idle waits for a full cold start. That's fine for an invoice queue and wrong for a login page. User-facing services should keep minReplicas at two or more.

#Security and cost limits

An autoscaler will happily turn an attack into capacity, and capacity into a bill.

maxReplicas is a blast-radius limit. During a flood of junk requests, an unbounded HPA scales the service, which scales the nodes, which may also exhaust database connections, IP addresses in the subnet, or third-party API quotas. Set maxReplicas from load tests ("30 replicas handle 3× our record peak") rather than a large round number, and alert when a workload sits at its maximum.

Here is the same simulator under a flood of junk requests. Switch between the two flood scenarios and compare dropped requests with Pod-minutes.

Denial of wallet is the cost version of denial of service: the attacker doesn't need to take you down if they can make you pay for 500 nodes. Layer hard ceilings below the HPA:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: shop-compute
  namespace: shop
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.memory: 120Gi
    pods: "120"

A ResourceQuota caps what one namespace can request, regardless of what any HPA asks for. Once a quota covers requests.cpu, every new Pod must declare a CPU request (or get one from a LimitRange), which conveniently enforces the HPA requirement too. Add node-level ceilings (Karpenter limits, Cluster Autoscaler --max-nodes-total, autoscaling-group maximums) and cloud budget alerts so a runaway shows up in hours, not at month end.

Beware attacker-controllable signals. Requests per second from the public internet, queue depth on a queue that anonymous users can write to, and custom metrics that any Pod can push are all inputs an attacker can drive. Rate-limit at the edge (the next lesson) before traffic reaches the metric, scale on authenticated work where possible, and protect the metrics pipeline itself: whoever can write to it, or edit HPA and ScaledObject resources through RBAC, can scale you to zero or to the ceiling.

#Practice

On a local cluster (kind or minikube), make an HPA scale a CPU-bound app, predict its decisions, then watch it fail without requests.

1. Install metrics-server.
2. Deploy registry.k8s.io/hpa-example (a PHP page that burns CPU) with a CPU request of 200m, and a Service for it.
3. Create an autoscaling/v2 HPA: 1 to 8 replicas, 50% CPU target, 60-second scale-down stabilisation.
4. Generate load and watch. When TARGETS first shows a value, compute the replica count the HPA should pick, and compare.
5. Stop the load and time the scale-down.
6. Remove the CPU request and observe the HPA.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl -n kube-system patch deployment metrics-server --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl top nodes   # works after about a minute

apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
spec:
  selector:
    matchLabels:
      app: php-apache
  template:
    metadata:
      labels:
        app: php-apache
    spec:
      containers:
        - name: php-apache
          image: registry.k8s.io/hpa-example
          ports:
            - containerPort: 80
          resources:
            requests:
              cpu: 200m
            limits:
              cpu: 500m
---
apiVersion: v1
kind: Service
metadata:
  name: php-apache
spec:
  selector:
    app: php-apache
  ports:
    - port: 80
---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 8
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 50
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 60

kubectl apply -f hpa-lab.yaml
kubectl run load --rm -it --image=busybox:1.36 --restart=Never -- \
  /bin/sh -c "while sleep 0.01; do wget -q -O- http://php-apache; done"
# in a second terminal:
kubectl get hpa php-apache --watch

NAME         REFERENCE               TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   cpu: 0%/50%    1         8         1          40s
php-apache   Deployment/php-apache   cpu: 248%/50%  1         8         1          75s
php-apache   Deployment/php-apache   cpu: 248%/50%  1         8         5          90s
php-apache   Deployment/php-apache   cpu: 62%/50%   1         8         5          2m
php-apache   Deployment/php-apache   cpu: 49%/50%   1         8         6          2m30s

kubectl patch deployment php-apache --type=json \
  -p '[{"op":"remove","path":"/spec/template/spec/containers/0/resources"}]'
kubectl get hpa php-apache
kubectl describe hpa php-apache | grep -A2 FailedGetResourceMetric

NAME         REFERENCE               TARGETS              MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   cpu: <unknown>/50%   1         8         1          9m
  Warning  FailedGetResourceMetric  20s  horizontal-pod-autoscaler  failed to get cpu utilization: missing request for cpu in container "php-apache" of Pod "php-apache-6d9f7b8c4-x2kqp"

#Key takeaways

• Scale out stateless services horizontally; use vertical scaling (VPA in Off mode) mainly to right-size requests.
• The HPA computes ceil(currentReplicas × current / target) per metric, takes the largest, ignores changes within 10%, and clamps to min and max; behavior sets how fast it moves.
• CPU utilisation is a percentage of the request: no request, no CPU autoscaling, and unrealistic requests give wrong decisions.
• Cluster Autoscaler or Karpenter add nodes when Pods are Pending; KEDA scales on event sources such as queue length and handles scale to zero.
• Pick signals that lead the load and that more Pods can actually fix; leave headroom for warm-up and cold starts.
• Bound everything: maxReplicas, ResourceQuotas, node pool limits and budgets protect against runaway scaling and denial of wallet, and edge rate limits stop attackers from driving the signal.