Design a URL shortener

A URL shortener turns https://shop.example.com/products/winter-boots?utm_source=newsletter&utm_campaign=oct into https://sho.rt/aZ3k9Qp and redirects anyone who opens the short link. It's a classic design problem because the core is small enough to finish, yet it touches ID generation, a read-heavy hot path, caching, asynchronous analytics and abuse. We'll design it with the same order of work as the first lesson.

#Requirements

Functional:

• Create a short link for a long URL, optionally with a custom alias (sho.rt/boots) and an expiry date.
• Redirect a short link to its long URL.
• Show the link's owner a click count over time.
• Out of scope: editing destinations, accounts beyond an API key, link previews.

Non-functional:

• Redirects are the product. p99 under 50 ms at our servers, and 99.99% availability for redirects (about 4 minutes of downtime a month). Creation can tolerate 99.9%.
• Links never break. A link that was created must redirect until it expires — for years.
• Short codes are short (7 characters) and hard to enumerate, so private links can't be discovered by counting.
• Analytics can lag by a minute and be approximate for unique visitors.

#Numbers

The estimation lesson's practice task worked these out: 100 million new links a month, a 100:1 read-to-write ratio, 500 bytes per link, kept for 10 years.

creates:   ~40/s average, ~200/s peak
redirects: ~4,000/s average, ~20,000/s peak
storage:   12 billion links × 500 B ≈ 6 TB over 10 years (18 TB with 3 replicas)
cache:     hottest 20% of a day's links ≈ 10 GB

The shape is clear: a modest write path, a hot read path that must be fast and available, and storage that grows steadily but predictably.

#API

POST /api/links
Authorization: Bearer <api key>
Idempotency-Key: 6f1c…
{"url": "https://shop.example.com/products/winter-boots?...", "alias": "boots", "expires_at": "2027-01-01T00:00:00Z"}

201 Created
{"code": "boots", "short_url": "https://sho.rt/boots", "expires_at": "2027-01-01T00:00:00Z"}

GET /aZ3k9Qp
302 Found
Location: https://shop.example.com/products/winter-boots?...
Cache-Control: private, max-age=60

The Idempotency-Key lets clients retry a timed-out create without making two links (see idempotency keys).

301 or 302? A 301 Moved Permanently is cached by browsers indefinitely, so repeat visits never reach us — less load, but we lose those clicks from analytics, and we can never change or disable the link for someone who's visited it once. A 302 Found (or 307) is fetched every time, unless we allow brief caching. Because analytics and the ability to disable malicious links are requirements, use 302 with a short max-age.

#Generating short codes

Seven characters from [0-9A-Za-z] give 62⁷ ≈ 3.5 trillion codes — 290 times the 12 billion links we expect. Four ways to produce them:

The simplest robust choice is random codes, inserted with the primary key as the uniqueness check. At 12 billion of 3.5 trillion codes, even after ten years a random code collides about 0.3% of the time, so one retry almost always suffices — and the database, not a "check then insert" race, decides.

const alphabet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

// randomCode returns n characters from alphabet, using crypto/rand so codes
// can't be predicted. rand.Int is unbiased, unlike taking a byte modulo 62.
func randomCode(n int) (string, error) {
	b := make([]byte, n)
	size := big.NewInt(int64(len(alphabet)))
	for i := range b {
		v, err := rand.Int(rand.Reader, size)
		if err != nil {
			return "", err
		}
		b[i] = alphabet[v.Int64()]
	}
	return string(b), nil
}

func (s *Store) Create(ctx context.Context, longURL string, expires *time.Time) (string, error) {
	for attempt := 0; attempt < 5; attempt++ {
		code, err := randomCode(7)
		if err != nil {
			return "", err
		}
		_, err = s.db.Exec(ctx,
			`INSERT INTO links (code, long_url, expires_at) VALUES ($1, $2, $3)`, code, longURL, expires)
		var pgErr *pgconn.PgError
		if errors.As(err, &pgErr) && pgErr.Code == "23505" { // unique_violation: try another code
			continue
		}
		return code, err
	}
	return "", errors.New("could not allocate a unique code")
}

Custom aliases go through the same insert; a conflict there means "alias taken" and goes back to the user. Reserve words such as api, admin and login up front. See unique ID generation for the other schemes.

#Data model and storage

CREATE TABLE links (
  code        text PRIMARY KEY,
  long_url    text NOT NULL,
  owner_id    bigint,
  created_at  timestamptz NOT NULL DEFAULT now(),
  expires_at  timestamptz,
  disabled_at timestamptz          -- set when a link is taken down for abuse
);

Every redirect is a lookup by primary key — a pure key-value access pattern. That fits a key-value store such as DynamoDB or Cassandra, which would scale storage indefinitely, and it fits PostgreSQL equally well at this size. 6 TB over ten years is within a single large PostgreSQL primary for the first several years; plan to hash-partition by code later, which works because no query ever spans codes. Two read replicas give the redirect path somewhere to read when the cache misses, and a standby for failover.

#The redirect path

This path carries 99% of traffic and owns the availability target, so it gets the most care.

Design points:

• Cache aggressively. Links are almost immutable, so a Redis cache with a long TTL gets a very high hit rate; 10 GB holds the hot set. On a takedown, delete the cache key along with setting disabled_at. See caching patterns.
• Optionally cache at the edge. A CDN can cache the 302 for a short time (say 60 seconds) for the hottest links, absorbing viral spikes. Because private keeps shared caches out, give the CDN its own instruction with a targeted header such as CDN-Cache-Control: max-age=60, which CDNs honour and browsers ignore. The edge then counts the clicks it serves (through its logs), or you accept those as uncounted.
• Reject junk cheaply. Scanners request random codes that don't exist; each would miss the cache and hit the database. Cache "not found" for a short TTL, and keep a Bloom filter of all existing codes in each redirect server's memory: a code the filter says is absent is definitely absent, so the server returns 404 without touching Redis or the database. At 12 billion codes and a 1% false-positive rate the filter needs about 14 GB, so in practice you'd keep one per partition or only for recent years.
• Stay up without dependencies. If Redis is down, read from replicas. If the click log is down, drop or buffer the click event, but still redirect. The redirect must never wait on analytics.

#Click analytics

Writing a row per click synchronously would put 20,000 writes per second on the database and couple redirect latency to it. Instead, the redirect service appends a click event — code, timestamp, country (from IP), referrer, user-agent class — to a log such as Kafka, without waiting. A stream processor consumes the log and maintains per-link, per-minute counts in an analytics store (ClickHouse, or counters in a table), which the owner's dashboard reads. See message queues vs event logs.

Unique visitors per link are expensive to count exactly (a set of visitor IDs per link). A HyperLogLog per link per day estimates them in about 12 KB with roughly 1% error, and sketches for different days or shards can be merged.

#Abuse and safety

Shorteners are popular with phishers because they hide the destination. Minimum defences:

• Check destinations against a malicious-URL list at creation, and re-check periodically.
• Rate-limit link creation per API key and per IP (rate-limiting algorithms).
• Support fast takedown: disabled_at plus cache deletion and a CDN purge, and a 302 with a short max-age so browsers don't keep sending people to a bad page.
• Never redirect to non-HTTP schemes such as javascript:.

#Practice

Two new product requirements arrive: (1) owners can change the destination of an existing short link (for example to fix a typo in a campaign already printed on posters), and changes must take effect within one minute; (2) a link-in-bio feature shows one public page per user listing their links, and some users have millions of followers.

Explain what changes in the redirect path, the caching, and the HTTP responses, and how you'd serve the link-in-bio page.

#Key takeaways

• Requirements and numbers show the shape: a small write path, a hot read path with a strict availability target, and steady storage growth.
• Use 302 with a short cache lifetime when you need analytics or the ability to change and disable links; 301 gives up both.
• Random 7-character base62 codes with the primary key as the uniqueness check are simple and safe; enumerable counters need scrambling.
• The redirect path reads by key through a cache, rejects unknown codes cheaply, and never waits on analytics.
• Count clicks asynchronously through a log, and estimate unique visitors with HyperLogLog.
• Plan for abuse from day one: destination checks, rate limits and fast takedown.