Graceful shutdown

Every deploy, scale-down and node drain stops your process. If it just exits, every request in flight fails, and with frequent deploys those errors add up. A graceful shutdown turns "stop" into "finish up, then stop."

#The sequence

1. Receive SIGTERM. Kubernetes, systemd and Docker send it first, then SIGKILL after a grace period (30 s by default in Kubernetes).
2. Fail readiness and keep serving for a few seconds. Load balancers and the Service's endpoint list take a moment to notice, and requests still arrive in the meantime.
3. Stop accepting new connections; let in-flight requests finish.
4. Stop background workers. Cancel their context and wait for them to finish their current job.
5. Close resources: database pools, message consumers, flushing telemetry.
6. Exit before the grace period ends.

ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
defer stop()

srv := &http.Server{Addr: ":8080", Handler: mux}
go func() {
	if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
		log.Fatal(err)
	}
}()

<-ctx.Done()                       // SIGTERM received
ready.Store(false)                 // /readyz now returns 503
time.Sleep(5 * time.Second)        // let load balancers notice

shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
_ = srv.Shutdown(shutdownCtx)      // stops listening, waits for active requests
workers.Stop()                     // cancel and wait for background jobs
db.Close()

#Details that bite

• Your shutdown budget must fit in the grace period. Readiness delay + request drain + cleanup has to be less than terminationGracePeriodSeconds, or SIGKILL cuts it off.
• Shutdown doesn't wait for hijacked connections like WebSockets. Track and close those yourself, and send clients a close frame so they reconnect elsewhere.
• Run as PID 1 correctly. If a shell script starts your binary, the signal goes to the shell. Use the exec form, ENTRYPOINT ["/app"], so your process receives it.
• Long jobs need to be resumable. Draining can't wait for a two-hour batch, so make the work checkpointed and idempotent, and let another instance pick it up.

Graceful shutdown is what makes rolling deploys invisible to users and makes stateless services cheap to replace.