News Feed System Design: Fan-Out, Celebrities and Ranking

The simple feed rebuilds every user's list from scratch each time they open the app. That work grows with users times opens times accounts followed, so the posts database is the first thing to struggle.

Picture a common setup. A regional social app in Lucknow lets people follow friends and local creators: street-food reviewers, poets, cricket coaches and small shops. The setup is made up, but the problem is real.

The team built the first feed in a week. On every open, the server finds everyone you follow, fetches their newest posts, sorts them by time and returns the top 20.

With 200,000 users, a feed loaded in under 300 milliseconds. Two years later the app has 4 million daily users, each opening the feed about 10 times a day.

That is 40 million feed loads a day, and about 2,300 a second at the evening peak. The posts database now runs at 90 percent CPU (processor use) every evening.

The slowest feeds take 3.5 seconds. This guide follows the fix: build feeds on read, then on write, then a mix of both, with ranking on top.

News feed system design is the design of the backend that collects posts from the accounts you follow and serves them as one ordered, endless list. It covers storage, fan-out, ranking and caching.

Four words come up again and again. Each gets a plain meaning and an example from the running app.

• Follow graph: the record of who follows whom. You follow 150 accounts, and a popular street-food creator has 1.5 million followers.
• Fan-out: one event spreading to many places. One post from that creator must reach 1.5 million feeds, so its fan-out is 1.5 million.
• Feed cache: a ready-made list of post ids for one user, kept in memory. An id is a short number that names a post, so the list stays small.
• Hydration: filling those ids with real content just before sending them, such as text, photos, author name and like count.

What the feed must do

• Show posts from accounts you follow, ranked, 20 at a time, with endless scrolling.
• Load the first page in under 200 milliseconds for most users, even at the evening peak.
• Show a new post to followers within a few seconds. A short delay is fine.
• Handle accounts with over a million followers without slowing everyone else down.
• Never show a deleted post, or a post from someone you blocked.

Notice the trade hidden in that list. Reads must be fast, but a short delay on new posts is acceptable. That small freedom makes the later designs possible.

Now the quick maths. 40 million loads a day is about 460 a second on average, and roughly five times that at peak. The app gets 400,000 new posts a day, about 5 a second.

So feed reads outnumber new posts by about 100 to 1. When reads dominate like this, you move work from read time to write time wherever you can.

Fan-out on read, also called the pull model, builds your feed at the moment you open it. Nothing is prepared in advance. The server pulls recent posts from every account you follow, then merges them.

Think of the panchayat noticeboard in a village. Every announcement is pinned once, in one place. Anyone who wants the news walks up and reads it.

Pinning is cheap, but every villager must walk over and scan the board. The team's first version worked exactly like this.

Each open looks up about 150 followed accounts. At 2,300 opens a second, that is about 345,000 author lookups a second, plus a big merge. The work repeats even if nobody posted anything new.

Pull has real strengths. A new post is stored once, so writing is cheap. Feeds are always fresh. And users who never open the app cost nothing at all.

Fan-out on write, also called the push model, copies each new post's id into every follower's feed cache when the post is made. Opening the feed then becomes one fast read of a ready list.

Now picture the village munadi, the drum-announcer. He walks house to house and calls out the news at every door. Each family hears it without moving, and he does all the walking.

Where the picture breaks: the munadi carries the whole message, but push copies only a small post id. And a real system can choose push or pull separately for each account.

A message queue sits between the post service and the workers. A message queue is a waiting line of events that workers pick up at their own pace. The post is confirmed at once, and delivery happens in the background.

The team uses Kafka for this queue. Kafka splits each stream of events, called a topic, into slices called partitions. Each event here is keyed by the author's id.

Kafka's documentation says events with the same key go to the same partition. Any reader of a partition sees its events in exactly the order they were written.

So workers see one author's post and that author's later delete in order. Each feed cache lives in Redis, a fast in-memory data store, as a sorted set: unique post ids kept in time order.

Push made reads fast. The evening peak now reads 2,300 small ready lists a second, and the posts database rests. The slowest feed loads fall from 3.5 seconds to about 90 milliseconds (made up, like all the app's numbers).

Use fan-out on write when reads far outnumber posts and most accounts have modest followings. Use fan-out on read for accounts with huge followings and for users who rarely open the app. Most large apps use both.

A 2010 research paper called Feeding Frenzy, from Yahoo! Research and Princeton University, studied exactly this choice. It decides push or pull separately for each pair of author and follower.

The rule compares two rates: how often the follower reads the feed, and how often the author posts. If the follower reads often enough compared with the author's posting rate, push. If not, pull.

The paper proves that, under its cost model, these per-pair choices give the lowest total cost. It also accepts slightly stale feeds, saying that is fine for many web apps.

The authors built it on PNUTS, a Yahoo! database, and tested it with real follow data crawled from a public social network: 200,000 followers and 67,921 authors. In their runs, the paper's hybrid beat both pure choices.

Neither pure choice won everywhere. With many posts, pushing everything cost 47 percent more load than pulling everything. With many reads, pulling everything cost 23 percent more than pushing.

A news feed handles celebrities by not pushing their posts at all. Their followers pull those posts at read time and merge them with the pushed feed. This mix is called the hybrid model.

The team learned this on a cricket match evening. Several big accounts posted within ten minutes, including the street-food creator with 1.5 million followers. Together they queued about 9 million feed inserts.

The fan-out workers handle about 15,000 inserts a second. So the queue took around 10 minutes to drain. Ordinary posts from friends waited behind it, and the fan-out lag alert went red.

The munadi has the same limit. He can walk every lane of one village, not every lane of the district for one announcement. Big district news goes on the noticeboard instead.

The team sets two rules. Accounts with more than 50,000 followers are never pushed; their posts stay in their own list of recent ids. Followers who have not opened the app in 30 days are skipped during push.

When a skipped user returns, the feed service rebuilds their feed once by pulling, like the old design. After that, push resumes for them, and the app stops writing millions of ids nobody reads.

Here is the read path after the change. Start at the feed service and see where its two kinds of candidates come from before ranking.

Ranking sits after fan-out and before hydration. Fan-out and pulling gather a few hundred candidate posts, ranking puts them in order, and hydration fills in only the 20 the user will see.

Keeping these apart matters. The fan-out pipeline cares about speed and delivery. Ranking cares about which posts a person will value, and its signals and training are a topic of their own.

1. 1

   Gather candidates

   Read up to 300 pushed ids from the feed cache, plus recent ids from each followed big account. Most users end up with 300 to 400 candidates.

2. 2

   Filter

   Drop deleted posts, blocked or muted authors, and posts the user already saw in this session.

3. 3

   Rank

   Send the candidate ids to the ranking service, which returns them in a new order. If it is too slow, sort by time instead.

4. 4

   Save the order

   Store the ranked list for this session for about 10 minutes. Page two reads from that saved list, so posts do not jump between pages.

5. 5

   Hydrate the first page

   Fetch text, photos, names and counts for the top 20 only, mostly from caches.

6. 6

   Return with a cursor

   Send the 20 posts and a cursor, a small marker that says where page two starts.

The fourth step fixes a quiet bug. If you rank again for every page, a post can move from page one to page two between requests. The user then sees it twice, or never.

Beginners can stop here: you have the whole design. The rest is for engineers who will build it: the code, the storage, measured memory numbers, and what breaks at scale.

Build both around Redis sorted sets. A sorted set holds unique members, here post ids, ordered by a number called the score.

The score here is the post time in milliseconds. When two posts share a score, Redis orders them by the bytes of the member text, so the order stays stable.

The fan-out worker

Here is the core of a worker. The helpers follower_count, follower_batches and is_active are made up; they read the follower lists. The r argument is a client from the redis Python package.

FEED_SIZE = 300          # keep the newest 300 ids per feed
AUTHOR_KEEP = 300        # and each author's newest 300 posts
CELEB_LIMIT = 50_000     # bigger accounts are pulled at read time

ADD_IF_FEED_EXISTS = """
if redis.call('EXISTS', KEYS[1]) == 0 then return 0 end
redis.call('ZADD', KEYS[1], ARGV[1], ARGV[2])
redis.call('ZREMRANGEBYRANK', KEYS[1], 0, -tonumber(ARGV[3]) - 1)
return 1
"""

def fan_out(post, r):
    own = f"posts:{post.author_id}"
    r.zadd(own, {post.id: post.created_ms})
    r.zremrangebyrank(own, 0, -AUTHOR_KEEP - 1)
    if follower_count(post.author_id) > CELEB_LIMIT:
        return                         # followers will pull this post
    pipe = r.pipeline(transaction=False)
    for batch in follower_batches(post.author_id, size=1000):
        for follower_id in batch:
            if not is_active(follower_id, days=30):
                continue               # rebuilt by pulling if they return
            pipe.execute_command("EVAL", ADD_IF_FEED_EXISTS, 1,
                                 f"feed:{follower_id}",
                                 post.created_ms, post.id, FEED_SIZE)
        pipe.execute()                 # one round trip per 1,000 followers

The first zadd records the post in the author's own list, which the read path and feed rebuilds need. The next line trims that list to 300, so it never grows forever.

The early return skips big accounts. Inside the loop, Redis runs a short script for each follower, written in Lua, a small language Redis can run on the server. Redis runs a script as one step, so no other command lands in the middle.

The worker sends the script text with every call, using EVAL. A Redis Cluster splits keys across several servers, and there the redis package's pipeline refuses EVALSHA, the cached-script command. EVAL works on one server and on a cluster.

If the follower's feed exists, the script adds the post id with ZADD and trims the feed to its newest 300 ids with ZREMRANGEBYRANK. If the feed is missing, the script skips it.

Sorted sets keep the lowest scores first, so trimming ranks 0 to minus 301 removes the oldest extras. In our test, after 310 posts a feed held 300 ids, from the 11th post to the newest.

A missing feed was never built, expired, or was lost with a cache node, and its next open rebuilds it by pulling. Without the check, one write recreates it holding a single post, and the read path never rebuilds it.

The pipeline sends many commands without waiting for each reply. Redis's documentation suggests sending big jobs in batches, such as 10,000 commands, because replies queue up in server memory. Here each batch covers 1,000 followers.

Retries are safe too. If a worker crashes and processes the same event again, ZADD only updates the existing member's score. In our test, a retried post still appeared once in each feed.

The read path

The helpers was_away, rebuild_by_pulling, followed_celebrities, drop_hidden, rank and hydrate are made up. The rank helper calls the ranking service and raises TimeoutError when it takes too long.

FEED_TTL = 30 * 24 * 3600      # a feed nobody opens for 30 days expires

def home_feed(user_id, r, limit=20):
    key = f"feed:{user_id}"
    if was_away(user_id, days=30) or not r.exists(key):
        rebuild_by_pulling(user_id, r)     # returning, new or lost feed
    r.expire(key, FEED_TTL)                # opening the feed keeps it alive
    pushed = r.zrange(key, 0, FEED_SIZE - 1, desc=True, withscores=True)
    pipe = r.pipeline(transaction=False)
    for celeb_id in followed_celebrities(user_id):
        pipe.zrange(f"posts:{celeb_id}", 0, 19, desc=True, withscores=True)
    pulled = [pair for rows in pipe.execute() for pair in rows]
    candidates = dict(pushed + pulled)     # one entry per post id
    candidates = drop_hidden(user_id, candidates)   # deleted, blocked, muted
    try:
        ordered = rank(user_id, candidates, timeout_ms=80)
    except TimeoutError:
        ordered = sorted(candidates, key=candidates.get, reverse=True)
    return hydrate(ordered[:limit])

The first check catches returning users, brand-new users and feeds lost with a cache node, and rebuilds them by pulling. The expire line gives the feed a 30-day timer that every open resets.

So a feed nobody opens for 30 days deletes itself and frees its memory. In our test, adding ids to a feed did not reset that timer; only an open did.

We also tested the gap the Lua check closes. A feed was deleted, then a new post arrived before the next open. The feed stayed missing, and the open rebuilt all 300 ids.

One case rebuilds more often than you might expect. Redis keeps no empty sorted sets, so a rebuild that finds nothing to push leaves no feed key behind.

A user who follows nobody yet, or only big accounts, therefore rebuilds on every open. In our test, three opens meant three rebuilds, so count these in your rebuild load.

The zrange calls use desc=True for newest first. Redis lists ZREVRANGE as deprecated since 6.2.0, replaced by ZRANGE with REV.

Yet redis-py 8.1.0 still sends ZREVRANGE for this call; we saw it in the server's MONITOR log. A comment in its source says this keeps desc working on older Redis servers. On Redis 7.0.15 both commands returned the same order.

The dict line merges both lists and keeps one entry per post id. If ranking takes over 80 milliseconds, the feed falls back to newest first, so users get a plainer order instead of an error.

Store each kind of data where it is cheapest to read: posts once in a durable database, the follow graph in both directions, and feed caches in memory. Durable means it survives crashes.

• Posts store. Each post is written once, keyed by post id, and sharded (split across machines) by that id.
• Follow graph. Two tables: the followers of each author, for fan-out, and the accounts each user follows, for pulling. Each is sharded by its first user id, so one lookup hits one machine.
• Author post lists. Each author's newest 300 post ids. The read path pulls from these for big accounts, and feed rebuilds use them too.
• Feed caches. One sorted set per active user, capped in size, in a Redis cluster. If one is lost, it can be rebuilt by pulling.
• Photos and videos. Kept in object storage and served through a CDN (a network of servers close to users), never inside the feed itself.

It depends mostly on how many ids each feed keeps, because Redis stores small and large sorted sets differently. On Redis 7.0.15 with default settings, a 300-id feed used about 42 KB.

On the same version, a 127-id feed used about 2.6 KB. That is about 16 times less memory for less than half the ids.

The reason is two memory layouts, which Redis calls encodings. Small sorted sets use listpack, a compact packed list. Bigger ones switch to skiplist, a faster structure with far more bookkeeping per member.

By default the switch happens above 128 members, or when a member is longer than 64 bytes. Redis 7.0 and later name these settings zset-max-listpack-entries and zset-max-listpack-value.

The table scales to 9 million feeds, because the app keeps one for each user who opened it in the last 30 days.

The 127 and 300 rows count everything Redis used, from its own memory counter, over 5,000 feeds. The others come from the MEMORY USAGE command, which reports one key's bytes.

These numbers come from Redis 7.0.15. Redis 8.6 changed how skiplist sorted sets are stored, and its release notes promise a substantial memory reduction for them.

The latest release as of 19 October 2026 is 8.10.2, which we could not measure. Expect smaller skiplist numbers there, and measure your own version before budgeting.

An earlier version of this guide guessed 60 bytes per id and 160 GB in total. On Redis 7.0.15 the measurement says about 380 GB for 300-id feeds, more than twice that guess.

The 128 trap

Capping feeds at exactly 128 does not keep them small. The worker adds first and trims after, so each feed briefly holds 129 ids and switches to skiplist.

In our Redis 7.0.15 test, trims never switched it back. A 128 cap ended as skiplist at about 16.8 KB, while a 127 cap stayed listpack at about 2.6 KB. Even a feed trimmed from 300 to 10 stayed skiplist.

Redis's memory page describes the switch up to the normal encoding, not a switch back. Treat the no-switch-back behaviour as undocumented as of 19 October 2026.

The Redis 8.10.2 source agrees. It converts back to listpack only when loading saved data, or for the result of a store command such as ZUNIONSTORE, which saves its answer as a new key.

Loading saved data matters here. Redis saves its data to disk as an RDB snapshot, and a replica (a copy server) loads one in a full sync. After a restart, and on a fresh replica, our 128-id feed came back as listpack at 3,120 bytes.

The next post tipped it back to skiplist. So the compact size after a restart does not last; budget for the skiplist size.

Raising the limit has a cost too. With the limit at 512, a 300-id feed stayed listpack at 6.2 KB. But each insert took about 15 to 40 microseconds of server time in our runs, against under 2.

So pick the cap on purpose. On Redis 7.0.15, keep 300 ids and budget about 380 GB, or keep 127 and about 24 GB. Pull older posts from author lists when someone scrolls past them.

Check it on your own Redis

Run these on a test server, not production. The first command fills a test feed with 300 ids, and each comment shows what Redis 7.0.15 printed for us.

# fill a test feed with 300 post ids (19 digits) and time scores
redis-cli EVAL "for i=1,300 do redis.call('ZADD', KEYS[1], 1760000000000+i, string.format('18%017d', i)) end return redis.call('ZCARD', KEYS[1])" 1 feed:test
redis-cli OBJECT ENCODING feed:test           # skiplist
redis-cli MEMORY USAGE feed:test SAMPLES 0    # about 40000 (bytes)
# trim to the newest 128: the layout does not switch back
redis-cli ZREMRANGEBYRANK feed:test 0 -129    # 172 removed
redis-cli OBJECT ENCODING feed:test           # still skiplist
# a fresh feed of 128 ids stays compact
redis-cli DEL feed:test
redis-cli EVAL "for i=1,128 do redis.call('ZADD', KEYS[1], 1760000000000+i, string.format('18%017d', i)) end return redis.call('ZCARD', KEYS[1])" 1 feed:test
redis-cli OBJECT ENCODING feed:test           # listpack
redis-cli MEMORY USAGE feed:test SAMPLES 0    # about 3100 (bytes)
redis-cli DEL feed:test

On Redis 7.0.15 the byte counts varied by a few hundred bytes between runs, and Redis 8.6 or later may print smaller skiplist numbers. Check the encodings: skiplist above 128 ids, listpack at or below it, no switch back after trims.

Most news feed failures come from the gap between a post and its copies. Copies arrive late, outlive the original, or pile up for the wrong people. Each failure needs a guard.

• Fan-out lag. A burst of big posts delays everyone. Guard: the celebrity threshold, plus an alert on the time from a post to its last feed write.
• Deleted posts. A deleted post's id still sits in many feeds. Guard: filter at read time, since hydration finds the post gone, and let a cleanup job remove leftovers later.
• Unfollows and blocks. Old ids from that author remain in one feed. Guard: filter at read time, then remove that author's ids from the one affected feed.
• Hot keys. One cache key is read far more than others, such as a big account's post list. Guard: keep extra copies, or a small local cache on each feed server.
• Lost cache node. Thousands of feeds vanish at once. Guard: rebuild each one by pulling on its next open, limit how many rebuilds run together, and let fan-out skip missing feeds.
• Duplicate events. A worker retries after a crash. Guard: zadd updates in place, so a retry adds no second copy.

Per-author order in Kafka has a limit. Kafka's operations guide warns that when data is partitioned by a hash of the key, adding partitions can shuffle that mapping.

Existing data does not move, so an author's post and its delete can land in different partitions around that change. The read-time filter still hides the deleted post, which is why it stays the main guard.

Three numbers sit on the team's main dashboard: fan-out lag, feed read time for the slowest one percent, and the share of feeds served without ranking. A rise in the last one means ranking is struggling.

Start from requirements and the read-to-write ratio, then walk from pull to push to hybrid. Finish with ranking, memory sizing and failure handling.

1. 1

   Ask for the key numbers

   Daily users, feed opens per user, posts per day, and average and largest follower counts. These numbers decide everything that follows.

2. 2

   Show the pull design first

   Draw the simple version and name its cost: every open merges posts from every followed account.

3. 3

   Move to push

   Add a queue, fan-out workers and per-user feed caches that hold ids only. Show why reads become cheap.

4. 4

   Break push with a celebrity

   Do the maths for one post to millions of followers. Then bring in the hybrid model and the skip for inactive users.

5. 5

   Add ranking and hydration

   Gather candidates, filter, rank with a timeout, hydrate one page, and return a cursor.

6. 6

   Size the cache and cover failures

   Estimate feed memory from the cap you chose, then cover deletes, unfollows, retries, lost cache nodes and the metrics you would watch.