# Graceful shutdown

> On SIGTERM, stop taking new work, finish what’s in flight, then exit — so deploys drop no requests.

Concept · Web services · https://levelcode.org/concepts/graceful-shutdown

Every deploy, scale-down and node drain stops your process. If it just exits, every request in flight fails, and with frequent deploys those errors add up. A **graceful shutdown** turns "stop" into "finish up, then stop."

## The sequence

1. **Receive `SIGTERM`.** Kubernetes, systemd and Docker send it first, then `SIGKILL` after a grace period (30 s by default in Kubernetes).
2. **Fail [readiness](https://levelcode.org/concepts/liveness-readiness)** and keep serving for a few seconds. Load balancers and the Service's endpoint list take a moment to notice, and requests still arrive in the meantime.
3. **Stop accepting new connections; let in-flight requests finish.**
4. **Stop background workers.** Cancel their [context](https://levelcode.org/concepts/context-cancellation) and wait for them to finish their current job.
5. **Close resources:** database pools, message consumers, flushing telemetry.
6. **Exit** before the grace period ends.

```go
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
defer stop()

srv := &http.Server{Addr: ":8080", Handler: mux}
go func() {
	if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
		log.Fatal(err)
	}
}()

<-ctx.Done()                       // SIGTERM received
ready.Store(false)                 // /readyz now returns 503
time.Sleep(5 * time.Second)        // let load balancers notice

shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
_ = srv.Shutdown(shutdownCtx)      // stops listening, waits for active requests
workers.Stop()                     // cancel and wait for background jobs
db.Close()
```

## Details that bite

- **Your shutdown budget must fit in the grace period.** Readiness delay + request drain + cleanup has to be less than `terminationGracePeriodSeconds`, or `SIGKILL` cuts it off.
- **`Shutdown` doesn't wait for hijacked connections** like WebSockets. Track and close those yourself, and send clients a close frame so they reconnect elsewhere.
- **Run as PID 1 correctly.** If a shell script starts your binary, the signal goes to the shell. Use the exec form, `ENTRYPOINT ["/app"]`, so your process receives it.
- **Long jobs need to be resumable.** Draining can't wait for a two-hour batch, so make the work checkpointed and [idempotent](https://levelcode.org/concepts/idempotent-consumers), and let another instance pick it up.

Graceful shutdown is what makes [rolling deploys](https://levelcode.org/concepts/deployment-strategies) invisible to users and makes [stateless services](https://levelcode.org/concepts/stateless-services) cheap to replace.