Leader election

Many designs need exactly one node to do something at a time: run a scheduler, own a partition, accept writes for a shard, or act as a replication primary. Leader election chooses that node, and chooses a new one when it fails.

#The usual mechanism: leases

1. Candidates try to acquire a lease (a lock with an expiry) in a strongly consistent store.
2. The winner is the leader. It renews the lease periodically — say every 5 seconds for a 15-second lease.
3. If the leader crashes or loses contact, it stops renewing, the lease expires, and another candidate acquires it.

Ready-made implementations:

• Kubernetes Lease objects with client-go's leaderelection package.
• etcd's election API (built on leases and revisions), Consul sessions, ZooKeeper ephemeral sequential nodes.
• Database advisory locks (pg_try_advisory_lock) for simple, single-database cases — the lock is released if the session dies.

#Choosing timings

• Lease duration: long enough to ride out brief network hiccups and pauses; short enough that failover is quick. 10–30 seconds is common.
• Renewal interval: a third of the lease or less, so a single missed renewal doesn't lose leadership.

#The dangerous case: two leaders

A leader can pause — a long garbage-collection stop, a VM migration, a frozen container — for longer than its lease. Meanwhile another node becomes leader. When the first one resumes, it still believes it's in charge. Checking "am I still leader?" before acting doesn't fully help: the pause can occur between the check and the action.

The fix is fencing: every lease comes with a monotonically increasing token, and the resource being protected rejects actions carrying an older token (leases and fencing tokens). Making the work itself idempotent is a second safety net.

#Underneath

The store holding the lease must itself be correct during failures, which is why it's built on consensus (consensus and Raft). Don't build leader election on a store that can lose acknowledged writes during failover.