# Leader election

> Pick exactly one node to do a job — and survive the moment two nodes both think it’s them.

Concept · Consistency & coordination · https://levelcode.org/concepts/leader-election

Many designs need exactly one node to do something at a time: run a scheduler, own a partition, accept writes for a shard, or act as a replication primary. Leader election chooses that node, and chooses a new one when it fails.

## The usual mechanism: leases

1. Candidates try to acquire a **lease** (a lock with an expiry) in a strongly consistent store.
2. The winner is the leader. It **renews** the lease periodically — say every 5 seconds for a 15-second lease.
3. If the leader crashes or loses contact, it stops renewing, the lease expires, and another candidate acquires it.

Ready-made implementations:

- Kubernetes `Lease` objects with client-go's `leaderelection` package.
- etcd's election API (built on leases and revisions), Consul sessions, ZooKeeper ephemeral sequential nodes.
- Database advisory locks (`pg_try_advisory_lock`) for simple, single-database cases — the lock is released if the session dies.

## Choosing timings

- **Lease duration:** long enough to ride out brief network hiccups and pauses; short enough that failover is quick. 10–30 seconds is common.
- **Renewal interval:** a third of the lease or less, so a single missed renewal doesn't lose leadership.

## The dangerous case: two leaders

A leader can **pause** — a long garbage-collection stop, a VM migration, a frozen container — for longer than its lease. Meanwhile another node becomes leader. When the first one resumes, it still believes it's in charge. Checking "am I still leader?" before acting doesn't fully help: the pause can occur between the check and the action.

The fix is **fencing**: every lease comes with a monotonically increasing token, and the resource being protected rejects actions carrying an older token ([leases and fencing tokens](https://levelcode.org/concepts/leases-and-fencing-tokens)). Making the work itself idempotent is a second safety net.

## Underneath

The store holding the lease must itself be correct during failures, which is why it's built on consensus ([consensus and Raft](https://levelcode.org/concepts/consensus-raft)). Don't build leader election on a store that can lose acknowledged writes during failover.