# Logs, metrics and traces

> Three kinds of telemetry and the question each answers: what happened, how much, and where the time went.

Concept · Web services · https://levelcode.org/concepts/logs-metrics-traces

Observability is your ability to answer new questions about production without shipping new code. It rests on three kinds of telemetry, each good at different questions.

| Signal | Answers | Shape | Cost grows with |
|---|---|---|---|
| **Metrics** | How much? How often? Is it getting worse? | Numbers over time, aggregated | Number of label combinations |
| **Logs** | What exactly happened in this case? | One event per line, ideally structured | Traffic volume |
| **Traces** | Where did this request spend its time? | A tree of timed spans across services | Traffic × sampling rate |

## Metrics: is something wrong?

Counters, gauges and histograms, such as `http_requests_total{route,status}` and `http_request_duration_seconds`. They're cheap to store and fast to query, and they're what you alert on. A common starting set is **RED** for services (rate, errors, duration) and **USE** for resources (utilization, saturation, errors). Record latency as a histogram so you can read percentiles. See [tail latency](https://levelcode.org/concepts/tail-latency).

**Watch label cardinality.** Labelling a metric with `user_id` creates one time series per user and can take down your metrics backend.

## Logs: what happened?

Write **structured** logs, as key–value pairs or JSON, so you can filter on fields:

```go
slog.Info("payment failed", "order_id", id, "provider", "stripe", "code", code, "trace_id", traceID)
```

Log decisions and failures, not every function call. Never log secrets, tokens or full card numbers. Logs are where sensitive data leaks most often.

## Traces: where did the time go?

A trace follows one request across services. Each hop records a **span** with a start time, a duration and attributes, and the trace context travels in headers (W3C `traceparent`). A trace shows that a 900 ms request spent 700 ms waiting on one slow query, or making [50 queries in a row](https://levelcode.org/concepts/n-plus-one-queries). Traces are usually **sampled** to control cost.

## Connecting them

- **Put the trace ID in every log line.** Then a slow trace leads you to its logs, and an error log leads you to its trace.
- **Use OpenTelemetry** to instrument once and send data to any backend.
- **Instrument in [middleware](https://levelcode.org/concepts/middleware)** so every route gets metrics and spans without per-handler code.

Metrics feed your [SLOs](https://levelcode.org/concepts/sli-slo-sla). When an alert fires, metrics tell you *that* something is wrong, traces tell you *where*, and logs tell you *why*.