# Design a URL shortener > Design a link shortener end to end: requirements and numbers, short-code generation, the redirect path at 20,000 requests per second, caching, click analytics and abuse. System Design roadmap · Case studies · 20 min read · https://levelcode.org/roadmaps/system-design/design-url-shortener A URL shortener turns `https://shop.example.com/products/winter-boots?utm_source=newsletter&utm_campaign=oct` into `https://sho.rt/aZ3k9Qp` and redirects anyone who opens the short link. It's a classic design problem because the core is small enough to finish, yet it touches ID generation, a read-heavy hot path, caching, asynchronous analytics and abuse. We'll design it with the same order of work as the first lesson. ## Requirements **Functional:** - Create a short link for a long URL, optionally with a custom alias (`sho.rt/boots`) and an expiry date. - Redirect a short link to its long URL. - Show the link's owner a click count over time. - Out of scope: editing destinations, accounts beyond an API key, link previews. **Non-functional:** - **Redirects are the product.** p99 under 50 ms at our servers, and 99.99% availability for redirects (about 4 minutes of downtime a month). Creation can tolerate 99.9%. - **Links never break.** A link that was created must redirect until it expires — for years. - **Short codes are short** (7 characters) and hard to enumerate, so private links can't be discovered by counting. - **Analytics can lag** by a minute and be approximate for unique visitors. ## Numbers The estimation lesson's practice task worked these out: 100 million new links a month, a 100:1 read-to-write ratio, 500 bytes per link, kept for 10 years. ```text creates: ~40/s average, ~200/s peak redirects: ~4,000/s average, ~20,000/s peak storage: 12 billion links × 500 B ≈ 6 TB over 10 years (18 TB with 3 replicas) cache: hottest 20% of a day's links ≈ 10 GB ``` The shape is clear: a modest write path, a hot read path that must be fast and available, and storage that grows steadily but predictably. ## API ```http POST /api/links Authorization: Bearer Idempotency-Key: 6f1c… {"url": "https://shop.example.com/products/winter-boots?...", "alias": "boots", "expires_at": "2027-01-01T00:00:00Z"} 201 Created {"code": "boots", "short_url": "https://sho.rt/boots", "expires_at": "2027-01-01T00:00:00Z"} ``` ```http GET /aZ3k9Qp 302 Found Location: https://shop.example.com/products/winter-boots?... Cache-Control: private, max-age=60 ``` The `Idempotency-Key` lets clients retry a timed-out create without making two links (see [idempotency keys](https://levelcode.org/concepts/idempotency-keys)). **301 or 302?** A `301 Moved Permanently` is cached by browsers indefinitely, so repeat visits never reach us — less load, but we lose those clicks from analytics, and we can never change or disable the link for someone who's visited it once. A `302 Found` (or `307`) is fetched every time, unless we allow brief caching. Because analytics and the ability to disable malicious links are requirements, use **302** with a short `max-age`. ## Generating short codes Seven characters from `[0-9A-Za-z]` give 62⁷ ≈ 3.5 trillion codes — 290 times the 12 billion links we expect. Four ways to produce them: | Approach | How | Problem | |---|---|---| | Hash and truncate | First 7 chars of base62(SHA-256(url)) | Collisions between different URLs need detection and a fallback | | Random code | 7 random base62 chars; insert; retry on conflict | Needs a uniqueness check (the primary key) and occasional retries | | Counter + base62 | Encode an incrementing integer | Sequential codes are enumerable unless scrambled | | Pre-allocated ranges | Each server reserves a block of the counter (say 10,000 IDs) | Same enumerability; needs a range allocator | The simplest robust choice is **random codes, inserted with the primary key as the uniqueness check**. At 12 billion of 3.5 trillion codes, even after ten years a random code collides about 0.3% of the time, so one retry almost always suffices — and the database, not a "check then insert" race, decides. ```go const alphabet = "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz" // randomCode returns n characters from alphabet, using crypto/rand so codes // can't be predicted. rand.Int is unbiased, unlike taking a byte modulo 62. func randomCode(n int) (string, error) { b := make([]byte, n) size := big.NewInt(int64(len(alphabet))) for i := range b { v, err := rand.Int(rand.Reader, size) if err != nil { return "", err } b[i] = alphabet[v.Int64()] } return string(b), nil } func (s *Store) Create(ctx context.Context, longURL string, expires *time.Time) (string, error) { for attempt := 0; attempt < 5; attempt++ { code, err := randomCode(7) if err != nil { return "", err } _, err = s.db.Exec(ctx, `INSERT INTO links (code, long_url, expires_at) VALUES ($1, $2, $3)`, code, longURL, expires) var pgErr *pgconn.PgError if errors.As(err, &pgErr) && pgErr.Code == "23505" { // unique_violation: try another code continue } return code, err } return "", errors.New("could not allocate a unique code") } ``` Custom aliases go through the same insert; a conflict there means "alias taken" and goes back to the user. Reserve words such as `api`, `admin` and `login` up front. See [unique ID generation](https://levelcode.org/concepts/unique-ids) for the other schemes. ## Data model and storage ```sql CREATE TABLE links ( code text PRIMARY KEY, long_url text NOT NULL, owner_id bigint, created_at timestamptz NOT NULL DEFAULT now(), expires_at timestamptz, disabled_at timestamptz -- set when a link is taken down for abuse ); ``` Every redirect is a lookup by primary key — a pure key-value access pattern. That fits a key-value store such as DynamoDB or Cassandra, which would scale storage indefinitely, and it fits PostgreSQL equally well at this size. 6 TB over ten years is within a single large PostgreSQL primary for the first several years; plan to **hash-partition by `code`** later, which works because no query ever spans codes. Two read replicas give the redirect path somewhere to read when the cache misses, and a standby for failover. ## The redirect path This path carries 99% of traffic and owns the availability target, so it gets the most care. ```sequence title: Redirecting a short link participants: Browser, Edge, Redirect service, Cache, Replica, Click log Browser -> Edge: GET /aZ3k9Qp Edge -> Redirect service: forward (cache miss at the edge) Redirect service -> Cache: GET link:aZ3k9Qp Cache --> Redirect service: miss Redirect service -> Replica: SELECT long_url, expires_at, disabled_at Replica --> Redirect service: row Redirect service -> Cache: SET link:aZ3k9Qp, TTL 1 h Redirect service -> Click log: append click event (async, non-blocking) Redirect service --> Browser: 302 Location: long URL ``` Design points: - **Cache aggressively.** Links are almost immutable, so a Redis cache with a long TTL gets a very high hit rate; 10 GB holds the hot set. On a takedown, delete the cache key along with setting `disabled_at`. See [caching patterns](https://levelcode.org/concepts/caching-patterns). - **Optionally cache at the edge.** A CDN can cache the 302 for a short time (say 60 seconds) for the hottest links, absorbing viral spikes. Because `private` keeps shared caches out, give the CDN its own instruction with a targeted header such as `CDN-Cache-Control: max-age=60`, which CDNs honour and browsers ignore. The edge then counts the clicks it serves (through its logs), or you accept those as uncounted. - **Reject junk cheaply.** Scanners request random codes that don't exist; each would miss the cache and hit the database. Cache "not found" for a short TTL, and keep a [Bloom filter](https://levelcode.org/concepts/bloom-filters) of all existing codes in each redirect server's memory: a code the filter says is absent is definitely absent, so the server returns 404 without touching Redis or the database. At 12 billion codes and a 1% false-positive rate the filter needs about 14 GB, so in practice you'd keep one per partition or only for recent years. - **Stay up without dependencies.** If Redis is down, read from replicas. If the click log is down, drop or buffer the click event, but still redirect. The redirect must never wait on analytics. ## Click analytics Writing a row per click synchronously would put 20,000 writes per second on the database and couple redirect latency to it. Instead, the redirect service **appends a click event** — code, timestamp, country (from IP), referrer, user-agent class — to a log such as Kafka, without waiting. A stream processor consumes the log and maintains per-link, per-minute counts in an analytics store (ClickHouse, or counters in a table), which the owner's dashboard reads. See [message queues vs event logs](https://levelcode.org/concepts/queue-vs-log). Unique visitors per link are expensive to count exactly (a set of visitor IDs per link). A [HyperLogLog](https://levelcode.org/concepts/hyperloglog) per link per day estimates them in about 12 KB with roughly 1% error, and sketches for different days or shards can be merged. ## Abuse and safety Shorteners are popular with phishers because they hide the destination. Minimum defences: - Check destinations against a malicious-URL list at creation, and re-check periodically. - Rate-limit link creation per API key and per IP ([rate-limiting algorithms](https://levelcode.org/concepts/rate-limiting-algorithms)). - Support fast takedown: `disabled_at` plus cache deletion and a CDN purge, and a 302 with a short `max-age` so browsers don't keep sending people to a bad page. - Never redirect to non-HTTP schemes such as `javascript:`. ## Practice Two new product requirements arrive: (1) owners can **change the destination** of an existing short link (for example to fix a typo in a campaign already printed on posters), and changes must take effect within one minute; (2) a **link-in-bio** feature shows one public page per user listing their links, and some users have millions of followers. Explain what changes in the redirect path, the caching, and the HTTP responses, and how you'd serve the link-in-bio page. **Solution** **(1) Editable destinations within a minute.** - *HTTP:* keep `302` and cap browser caching at `Cache-Control: private, max-age=60`, so a browser that already visited picks up the change within a minute. Never use `301` for editable links: browsers may cache it indefinitely and you can't take it back. - *Edge:* any CDN caching of redirects must also use a TTL of 60 seconds or less, or the update must trigger a purge of that path through the CDN's API. - *Application cache:* on update, write the new destination to the database, then delete `link:` from Redis after the commit. To guard against the read/write race re-caching the old value, keep the Redis TTL short for recently edited links, or delete again a few seconds later. - *Data:* store a version or `updated_at`, and record edits in an audit table — a destination change is a phishing vector if an account is taken over, so notify the owner by email. **(2) Link-in-bio pages for very popular users.** - The page is the same for every visitor and changes only when the user edits their links, so **render it once and cache it at the CDN** with a short TTL (say 60 seconds) and purge on edit. A user with millions of followers then costs the origin roughly one request per minute per edge location. - The data is a small list: `bio_links(user_id, position, code, title)`, read by `user_id`. Cache the rendered page or its JSON in Redis keyed by user, invalidated on edit. - Clicks from the page are ordinary short-link redirects, so analytics needs no change. Treat a sudden spike on one page as a [hot key](https://levelcode.org/concepts/hot-partitions): the CDN absorbs it, and an in-process cache in the page service covers edge misses. ## Key takeaways - Requirements and numbers show the shape: a small write path, a hot read path with a strict availability target, and steady storage growth. - Use `302` with a short cache lifetime when you need analytics or the ability to change and disable links; `301` gives up both. - Random 7-character base62 codes with the primary key as the uniqueness check are simple and safe; enumerable counters need scrambling. - The redirect path reads by key through a cache, rejects unknown codes cheaply, and never waits on analytics. - Count clicks asynchronously through a log, and estimate unique visitors with HyperLogLog. - Plan for abuse from day one: destination checks, rate limits and fast takedown. ## Quiz 1. Why does this design use 302 redirects instead of 301? A. 302 responses are smaller B. Browsers cache 301s indefinitely, which loses click analytics for repeat visits and makes links impossible to change or disable for those users C. 301 is deprecated D. 302 redirects are faster for browsers to follow **Answer: B.** A permanent redirect is the browser's to keep. When you need every click and the ability to take a link down, a temporary redirect with a short max-age keeps control on the server. 2. Short codes are 7 random base62 characters. What is the safest way to guarantee uniqueness? A. SELECT to check the code is free, then INSERT B. Insert with the code as the primary key and generate a new code if the insert hits a unique violation C. Use the first 7 characters of an MD5 hash of the URL, which can't collide D. Keep a list of used codes in each server's memory **Answer: B.** Check-then-insert is a race between concurrent creators. Letting the database's unique constraint decide makes the check and the write one atomic step; collisions are rare enough that a retry is cheap. 3. Scanners request millions of random codes that don't exist. What stops them reaching the database? A. A longer cache TTL for existing links B. A Bloom filter of existing codes, plus caching 'not found' briefly, so absent codes are rejected in memory C. Switching to 301 redirects D. Adding more read replicas **Answer: B.** Unknown codes always miss a cache of existing links. A Bloom filter answers 'definitely absent' with no false negatives, so those requests get a 404 without any database lookup. 4. Why are click events written to a log asynchronously rather than inserted into the database during the redirect? A. Databases can't store timestamps accurately B. It keeps redirect latency and availability independent of analytics, and turns 20,000 writes per second into a stream processed in batches C. Logs are more accurate than databases D. Browsers require the response before any write **Answer: B.** The redirect is the product; analytics may lag by a minute. Decoupling them means an analytics outage can't break redirects, and aggregation happens off the hot path.