# Design a news feed > Design a social home timeline: fan-out on write versus read, the celebrity problem and the hybrid that solves it, timeline caches, hydration, pagination and ranking. System Design roadmap · Case studies · 21 min read · https://levelcode.org/roadmaps/system-design/design-news-feed A news feed — the home timeline of Twitter/X, Instagram, LinkedIn or Mastodon — shows each user recent posts from the accounts they follow. It looks like a single query: "posts by people I follow, newest first". At hundreds of millions of users, that query becomes one of the most interesting trade-offs in system design, because the work can be done when a post is **written** or when a feed is **read**, and the right answer is "both, depending on who posted". ## Requirements **Functional:** users publish posts (text plus optional images); follow and unfollow accounts; open a home feed of posts from accounts they follow, newest first, paginated by scrolling. Ranking by relevance is a stretch goal. Out of scope: replies, direct messages, search. **Non-functional:** - Feed loads in under 200 ms at p99; it's the most-used screen in the app. - A new post appears in followers' feeds within a few seconds (but not instantly — eventual consistency is fine here). - High availability: a stale feed is far better than an error. - Follower counts are wildly uneven: most accounts have a few hundred followers; a few have tens of millions. ## Numbers ```text daily active users: 200 million posts: 20% of users post once a day → 40M posts/day ≈ 400/s avg, ~2,000/s peak feed opens: 10 per user per day → 2B/day ≈ 20,000/s avg, ~100,000/s peak average follows: 200 accounts average followers: 200 — but the top accounts have 50–100 million ``` Feeds are read 50 times more often than posts are written, and reads have the strict latency target. That pushes toward precomputing feeds, as long as precomputing doesn't explode for the accounts with enormous followings. ## Data model ```sql -- Posts, partitioned by author: "this author's recent posts" is one partition. CREATE TABLE posts ( post_id bigint PRIMARY KEY, -- Snowflake-style: time-ordered author_id bigint NOT NULL, body text NOT NULL, media_keys text[], created_at timestamptz NOT NULL, deleted_at timestamptz ); CREATE INDEX posts_by_author ON posts (author_id, post_id DESC); -- The follow graph, stored in both directions so each lookup is one partition. CREATE TABLE following (follower_id bigint, followee_id bigint, PRIMARY KEY (follower_id, followee_id)); CREATE TABLE followers (followee_id bigint, follower_id bigint, PRIMARY KEY (followee_id, follower_id)); ``` Time-ordered post IDs (see [unique ID generation](https://levelcode.org/concepts/unique-ids)) make "newest first" a sort by ID and give natural pagination cursors. The graph is denormalised into two tables because both directions are hot: "who do I follow?" when reading a feed, "who follows me?" when distributing a post. A graph database isn't needed; these are one-hop lookups. ## Option 1: fan-out on read Build the feed when the user opens it: look up the ~200 accounts they follow, fetch each account's latest posts, and merge them newest first. ```text feed(user) = top 20 of merge( recent_posts(a) for a in following(user) ) ``` - **Writes are trivial:** a post is one insert. - **Reads are expensive:** 200 lookups per feed open, merged, at 100,000 opens per second — 20 million post-list lookups per second at peak, with latency set by the slowest of 200 parallel reads ([tail latency](https://levelcode.org/concepts/tail-latency)). ## Option 2: fan-out on write When someone posts, push the post ID into a precomputed **timeline** for each of their followers. Reading a feed is then a single lookup. ```text on post(author, post_id): for f in followers(author): timeline[f].prepend(post_id); trim to 800 entries feed(user) = timeline[user][0:20] ``` Timelines live in an in-memory store: a Redis list or sorted set of post IDs per user, capped at a few hundred entries. 200 million timelines × 800 IDs × 8 bytes ≈ 1.3 TB of IDs — large but spread across a Redis cluster, and you'd only keep timelines for users active in the last few weeks. - **Reads are cheap:** one list read, then fetch the 20 posts. - **Writes multiply:** an average post becomes 200 timeline inserts — 2,000 posts per second become 400,000 inserts per second. Done asynchronously by workers, that's fine. - **Celebrities break it.** An account with 80 million followers turns one post into 80 million inserts. Even at a million inserts per second, the last followers see it more than a minute later, and the fan-out queue backs up for everyone else. ```sequence title: Fan-out on write for an ordinary author participants: Author, Post service, Database, Fan-out queue, Fan-out worker, Timeline cache Author -> Post service: POST /posts Post service -> Database: INSERT post + outbox row Post service --> Author: 201 created Post service -> Fan-out queue: post.created (via the outbox) Fan-out queue -> Fan-out worker: deliver Fan-out worker -> Database: followers(author), in pages of 1,000 Fan-out worker -> Timeline cache: LPUSH + LTRIM for each follower ``` The outbox ([transactional outbox](https://levelcode.org/concepts/transactional-outbox)) guarantees every committed post is fanned out, and pushing IDs (not whole posts) keeps the timelines small and lets edits and deletions take effect everywhere at once. ## Option 3: the hybrid Use fan-out on write for the vast majority of authors and fan-out on read for the few with huge followings: - Accounts above a follower threshold (say 1 million) are marked **celebrities**. Their posts are *not* pushed. - Reading a feed: take the user's precomputed timeline, **plus** the recent posts of the celebrities they follow (usually a handful), merge, and return the top 20. ```go func (s *Feed) Home(ctx context.Context, userID int64, limit int) ([]Post, error) { // Pushed timeline: post IDs from ordinary authors, newest first. ids, err := s.timelines.Range(ctx, userID, 0, 200) if err != nil { return nil, err } // Pulled: the few celebrities this user follows. Their recent post IDs are cached hot. for _, celeb := range s.graph.CelebritiesFollowedBy(ctx, userID) { ids = append(ids, s.posts.RecentIDs(ctx, celeb, 50)...) } // Post IDs are time-ordered, so newest first is a descending sort. slices.SortFunc(ids, func(a, b int64) int { return cmp.Compare(b, a) }) ids = slices.Compact(ids) return s.posts.Hydrate(ctx, ids[:min(limit, len(ids))]) // skips deleted or hidden posts } ``` Celebrity posts are read by millions of people, so "recent posts of account X" is itself a [hot key](https://levelcode.org/concepts/hot-partitions) — cache it in Redis and in process memory for a few seconds. This is the design Twitter described publicly, and variations of it run most large feeds. ## Hydration, deletions and privacy The timeline holds only IDs. **Hydration** turns 20 IDs into renderable posts with author names, avatars, like counts and media URLs, using multi-get lookups against a post cache and a user cache. Keeping content out of the timelines has useful consequences: - **Deletes and edits** update one post record; every timeline that references it shows the change, and hydration skips deleted posts. - **Blocks, mutes and private accounts** are applied during hydration, so they take effect immediately even though timelines were precomputed. - **Unfollows** can leave stale IDs in a timeline; filter them during hydration by checking the follow relation, and let them age out. Counts such as likes are themselves a write hot spot on viral posts; keep them as sharded counters or in a write-behind cache, and show approximate values ("12.4K"). ## Pagination and ranking Scrolling needs a cursor, not an offset: new posts arrive at the top while the user scrolls, and `OFFSET` would skip or repeat items. Return the last post ID seen and ask for "IDs less than this" next time — [keyset pagination](https://levelcode.org/concepts/keyset-pagination). ```http GET /feed?limit=20 → posts …, next_cursor = 1849302910228 GET /feed?limit=20&before=1849302910228 ``` A **ranked** feed changes the read step, not the architecture: gather a few hundred candidates (the pushed timeline plus celebrity posts plus, perhaps, recommended posts), score each with a model (predicted engagement, freshness, relationship strength), and return the top results. Because scores change, cursors for ranked feeds usually encode a snapshot ID — the scored list is cached for a few minutes so the next page continues the same ranking. ## Degrading gracefully - **Inactive users:** don't fan out to users who haven't opened the app in 30 days. When one returns, rebuild their timeline on read (fan-out on read once), then resume pushing. - **Timeline cache loss:** a lost Redis shard means those users' timelines are rebuilt on read. Rebuilding is expensive, so rate-limit rebuilds and serve a degraded feed (celebrity and popular posts) meanwhile. - **Fan-out backlog:** if workers fall behind, feeds are slightly stale, not broken. Alert on queue lag, and prioritise fan-out to recently active users. See [graceful degradation](https://levelcode.org/concepts/graceful-degradation). ## Practice Using the hybrid design, walk through two events: 1. An account with 80 million followers, flagged as a celebrity, posts. What happens on write, and what does a follower's next feed request do? 2. A user follows a new account with 300 followers (not a celebrity) that posted 15 times this week. Today, those posts don't appear in the user's feed until the account posts again. Fix it. Also explain what happens when an ordinary account crosses the 1-million-follower threshold. **Solution** **1. A celebrity posts.** The post is inserted into `posts` with its outbox row, and the author gets `201` immediately. The fan-out worker sees that the author is a celebrity and does **not** push the ID to 80 million timelines; it only adds the ID to the cached "recent posts" list for that author (and may invalidate the author's cached recent-posts entry). When a follower next opens the feed, the read path fetches their pushed timeline plus the recent posts of the few celebrities they follow — served from Redis and in-process caches, since that list is read millions of times — merges by ID, hydrates the top 20 and returns them. The post shows up on the follower's next feed load, with no fan-out delay at all. **2. Following an account.** Following changes the graph but not the timeline. Fix: on follow, enqueue a **backfill** job: fetch the followee's most recent posts (say the last 50 or the last week) and merge their IDs into the follower's timeline, then trim to the cap. Because IDs are time-ordered, the merge places them correctly. Unfollow does the reverse lazily: hydration filters out posts from accounts the user no longer follows, and they age out. **3. Crossing the threshold.** When an account passes 1 million followers, flag it as a celebrity. From then on, its new posts are pulled rather than pushed. Its older posts are already in followers' timelines and age out naturally. To avoid flapping around the threshold, use hysteresis — become a celebrity at 1 million, stop being one only below 800,000. In the opposite direction, when an account loses celebrity status, its future posts are pushed normally, and its recent posts stay available through the pull path until they're older than anything in followers' timelines. ## Key takeaways - Fan-out on read makes writes cheap and reads expensive; fan-out on write precomputes timelines so reads are one lookup. - Celebrities make fan-out on write explode; the hybrid pushes ordinary authors' posts and pulls the few huge accounts at read time. - Timelines hold post IDs only; hydration applies deletes, edits, blocks and privacy at read time. - Use the transactional outbox so every committed post is fanned out, and cache celebrity post lists as hot keys. - Paginate with cursors on time-ordered IDs; ranking changes the read step, not the storage. - Skip fan-out for inactive users and rebuild their timelines on demand. ## Quiz 1. Why does pure fan-out on write fail for accounts with tens of millions of followers? A. Their posts are too large to store B. One post becomes tens of millions of timeline inserts, delaying delivery and backing up the fan-out pipeline for everyone C. Followers of celebrities read their feeds less often D. Redis can't store that many lists **Answer: B.** Write amplification is proportional to follower count. At 80 million followers, a single post takes minutes to fan out and consumes capacity needed for everyone else's posts. 2. In the hybrid design, how do a celebrity's posts reach a follower's feed? A. They are pushed to every follower's timeline like other posts B. They are emailed to followers C. At read time, the feed service fetches the recent posts of the celebrities the user follows and merges them with the pushed timeline D. They only appear in search results **Answer: C.** Celebrity posts are pulled on read, which is cheap because each user follows only a few celebrities and those post lists are heavily cached. 3. Why do precomputed timelines store post IDs rather than full post content? A. IDs are required by Redis B. It keeps timelines small, and lets deletes, edits, blocks and privacy changes apply at hydration time without rewriting millions of timelines C. Content can't be cached D. It makes posts load in a random order **Answer: B.** With IDs, the post record is the single source of truth. A deletion is one update, and every timeline that references the post reflects it the next time it's hydrated. 4. New posts keep arriving while a user scrolls. Why use a cursor ("posts older than ID X") instead of OFFSET for the next page? A. OFFSET is not supported by Redis B. With OFFSET, new posts at the top shift every item down, so the next page repeats posts; a cursor continues exactly where the user left off C. Cursors make posts load faster for the first page D. OFFSET can only be used with ranked feeds **Answer: B.** Offsets count from the current top of a moving list. Keyset pagination anchors on the last item seen, so inserts above it don't affect the next page.