Concept · Web services
Logs, metrics and traces
Three kinds of telemetry and the question each answers: what happened, how much, and where the time went.
Observability is your ability to answer new questions about production without shipping new code. It rests on three kinds of telemetry, each good at different questions.
| Signal | Answers | Shape | Cost grows with |
|---|---|---|---|
| Metrics | How much? How often? Is it getting worse? | Numbers over time, aggregated | Number of label combinations |
| Logs | What exactly happened in this case? | One event per line, ideally structured | Traffic volume |
| Traces | Where did this request spend its time? | A tree of timed spans across services | Traffic × sampling rate |
Metrics: is something wrong?
Counters, gauges and histograms, such as http_requests_total{route,status} and http_request_duration_seconds. They're cheap to store and fast to query, and they're what you alert on. A common starting set is RED for services (rate, errors, duration) and USE for resources (utilization, saturation, errors). Record latency as a histogram so you can read percentiles. See tail latency.
Watch label cardinality. Labelling a metric with user_id creates one time series per user and can take down your metrics backend.
Logs: what happened?
Write structured logs, as key–value pairs or JSON, so you can filter on fields:
slog.Info("payment failed", "order_id", id, "provider", "stripe", "code", code, "trace_id", traceID)Log decisions and failures, not every function call. Never log secrets, tokens or full card numbers. Logs are where sensitive data leaks most often.
Traces: where did the time go?
A trace follows one request across services. Each hop records a span with a start time, a duration and attributes, and the trace context travels in headers (W3C traceparent). A trace shows that a 900 ms request spent 700 ms waiting on one slow query, or making 50 queries in a row. Traces are usually sampled to control cost.
Connecting them
- Put the trace ID in every log line. Then a slow trace leads you to its logs, and an error log leads you to its trace.
- Use OpenTelemetry to instrument once and send data to any backend.
- Instrument in middleware so every route gets metrics and spans without per-handler code.
Metrics feed your SLOs. When an alert fires, metrics tell you that something is wrong, traces tell you where, and logs tell you why.