Instagram System Design: Photos, Stories and the Feed

A photo app slows down because images are large and every scroll asks for dozens of them at once. When one server stores, resizes and sends every image, it runs out of disk, memory and network together. Instagram system design is about splitting that work, so each part can grow on its own.

Picture a common setup. A regional photo-sharing app in India lets people post photos and share stories that vanish after 24 hours. The setup is made up, but the problems are real.

One server takes each upload, saves the file on its own disk and writes a row in a database. The same server sends the photo back whenever someone scrolls. It works well for 20,000 users.

Then a festival weekend arrives, and uploads jump five times in two days. Four things break.

• Uploads time out on slow mobile networks, because the server holds each connection open while it resizes the photo.
• Feeds load slowly, because the server sends 3 MB originals to phones that show them only 400 pixels wide.
• The disk fills up on Sunday night. Nobody planned where the ten millionth photo would live.
• Friday's stories still show on Sunday, because the one cleanup script crashed and nobody noticed.

This guide follows the team's redesign of that app, one part at a time, and checks each claim against a paper, a doc or a test we ran.

Instagram system design is the task of building a photo-sharing app that stores a huge number of images and shows them to millions of people quickly. It covers upload, storage, delivery, stories and the home feed.

It works well as an interview question because it mixes big files, small database rows, heavy caching and a feature with a deadline built in. Five words come up again and again.

The core features are posting a photo with a caption, following people, seeing a feed of their posts, and sharing stories that disappear after 24 hours. Likes and comments come next, and search can wait.

The requirements that shape the design are about speed and safety, not features. The team agreed on these five.

• Feeds and profiles load fast, even on a cheap phone with a weak signal.
• An uploaded photo is never lost, even if a disk or a machine dies.
• A new post may take a few seconds to appear for followers.
• A story stops showing to viewers 24 hours after it was posted.
• Private accounts never leak photos to people who are not approved followers.

Next, size the load with rough numbers. This is a back-of-the-envelope estimate: quick sums that show which part will hurt first. The table shows the sums for an ordinary day.

Look at the ratio. There are about 4.2 lakh uploads a day but 14.4 crore image views. That is about 340 reads for every write, so reading must be cheap first and writing must be safe second.

The phone uploads the photo straight to object storage using a short-lived signed link, then tells your API, the app's own server, that it is done. A background worker, a program that runs jobs away from the user, makes the smaller sizes, and only then does the post go live.

A signed link, often called a presigned URL, is a web address that allows an upload to one key for a limited time. The phone needs no storage password. Amazon S3 documents this pattern, and other object stores offer the same idea.

This fixes the first broken thing. The slow part, pushing 3 MB over a weak network, now happens between the phone and the storage service. The API servers handle only small requests that finish quickly.

Follow the arrows below in order. Each box is one message, and its detail line says who talks to whom.

The job queue is a waiting line of tasks between the API and the workers. If 50,000 photos arrive in an hour, the jobs wait in line instead of timing out. The team adds workers when the line grows long.

The confirm call carries an upload ID that the phone got in the first step. If the network drops and the phone retries, the API sees the same ID and does not create a second post.

Store the photo bytes in object storage, keep a small metadata row in a database, and serve images through a CDN. Each layer does one job, so each can grow on its own.

What Facebook learned from storing one file per photo

Facebook's 2010 Haystack paper explains why this split matters. Its first photo store kept one file per photo on network storage, and reading one photo often took more than 10 disk operations.

After tuning, it still generally took 3: one for the folder's metadata, one for the file's record, and one for the photo itself. Only the last one was useful work.

Haystack packed many photos into a few huge files and kept a small index in memory, about 10 bytes per photo. A read then needed at most one disk operation.

You will not build Haystack for this app. Managed object storage already handles file layout for you. The lesson that stays is simpler: looking up where a photo lives must cost far less than reading the photo.

Why store each photo in several sizes?

Each phone needs only enough pixels to fill its screen, so sending the original wastes data and time. The Haystack paper says Facebook generated and stored four sizes of every uploaded photo.

The team's worker makes three sizes: 150 pixels wide for the profile grid, 640 for the feed, and 1080 for full screen. The engineer section shows the worker and its output.

How does the CDN fit in?

The CDN answers most image requests, and only misses reach your own storage. Think of a busy shop with a front window and a stockroom upstairs.

Where the picture breaks: a shop has one window, but a CDN has thousands of separate windows in many cities. A photo can be in Mumbai's copy and missing from Guwahati's.

Pause and predict. Facebook's photo traffic passed through the phone or browser's own cache, then CDN edge servers, then a middle cache, then the Haystack store. What share of requests do you think reached the store?

So about 90 percent of requests never reached storage. The misses were not spread evenly: for the least popular group of photos, Haystack served almost 80 percent of requests.

The Haystack paper calls these rarely viewed photos the long tail, and says their requests typically miss the CDN. That is why the store must still serve old photos well.

One more habit makes the CDN work well. The app never changes a file in place. If a user edits a photo, the worker saves it under a new key, and the post row points to the new key.

Because a key's content never changes, the CDN can keep each file for a very long time. Haystack also never overwrote a photo in place, but it appended the new version under the same key.

Stories disappear because the app hides any story older than 24 hours every time someone reads them. Archiving or deleting the actual file happens later, in a separate cleanup step that can run late without anyone seeing.

Instagram itself shows why expiry is about viewers, not files. Its December 2017 announcement said expired stories save automatically to a private archive, which the owner can turn off.

The same post introduced highlights, collections of stories shown on the profile until the owner deletes them. So 'gone after 24 hours' means hidden from viewers, not necessarily deleted.

Go back to the shop. The shutter comes down at closing time, even if staff take the display apart an hour later. No customer sees yesterday's offers through the glass.

The read filter is the shutter, and the cleanup job is the staff. The first design had only the staff, so when the script crashed, Friday's stories stayed in the window.

Here is the filter that runs on every story read. Each story is a dictionary with its creation time and the set of people its author hid it from.

from datetime import datetime, timedelta, timezone

STORY_LIFETIME = timedelta(hours=24)

def visible_stories(stories, viewer_id, now):
    cutoff = now - STORY_LIFETIME
    shown = []
    for story in stories:
        if story["created_at"] <= cutoff:
            continue  # expired, even if the file still exists
        if viewer_id in story["hidden_from"]:
            continue
        shown.append(story)
    return sorted(shown, key=lambda s: s["created_at"])

posted = datetime(2026, 10, 18, 13, 0, tzinfo=timezone.utc)
story = {"id": "s1", "created_at": posted, "hidden_from": {42}}
for age in ["23:59:59", "24:00:00", "24:00:01"]:
    h, m, s = map(int, age.split(":"))
    now = posted + timedelta(hours=h, minutes=m, seconds=s)
    print(age, len(visible_stories([story], 7, now)), len(visible_stories([story], 42, now)))

We ran it on Python 3.13.16. Each line prints the story's age, then how many stories viewer 7 sees, then how many viewer 42 sees.

23:59:59 1 0
24:00:00 0 0
24:00:01 0 0

What just happened? At one second under 24 hours, viewer 7 sees the story, and viewer 42 never does because the author hid it. At exactly 24 hours, the story is hidden for everyone, whatever the cleanup job has done.

Passing now as an argument makes the boundary testable without waiting a day. Keep every time in UTC, one world clock, so viewers in different time zones agree.

Why not let the storage service delete stories on time?

Storage expiry rules are too coarse and too slow to enforce a 24-hour promise. S3 lifecycle rules based on age count whole days, round the expiry to midnight UTC the next day, and remove objects asynchronously, sometimes with a delay.

So the team uses an expiry rule only as a safety net, set to two days on the stories area. The engineer section works out how late that net can be.

Hiding a story in the tray does not stop someone who saved the file's link. That is why story files are served through short-lived signed links, and why the CDN must not keep them past the story's life.

A story saved to a highlight must leave the stories area before the safety net catches it. The app copies that file into the posts area, where no expiry rule applies.

How does the stories tray know whose stories to show?

The tray checks which people you follow have a story from the last 24 hours. The team keeps one small cache entry per author, holding the time of their newest story, set to expire 24 hours later.

When a viewer opens the app, the API looks up that cache for the people they follow. Only authors with a live entry appear in the tray. A second small set records which stories the viewer has seen, so the app can dim those rings.

The home feed is a separate service that returns an ordered list of post IDs for each user. The photo system then turns those IDs into captions, like counts and image links.

Building that list is the hard part of the feed. You can copy each new post into every follower's list when it is posted, or gather posts when someone opens the app. Accounts with millions of followers usually need a mix of both.

Those choices are covered in our guide on news feed system design. Here, the team needs only the step after the list is ready.

1. 1

   Get the next page of IDs

   The feed service returns, say, 20 post IDs for the viewer, newest or best first.

2. 2

   Fetch the post rows in one batch

   The Post API asks the cache for all 20 rows at once. Only the missing ones go to the database shards.

3. 3

   Drop posts that should not show

   Remove deleted posts, posts still processing, and posts from accounts that blocked the viewer.

4. 4

   Build the image links

   For each post, pick the size that fits the screen and build a CDN link from the storage key.

5. 5

   Send one small reply

   The app gets captions and links, then fetches the images from the CDN on its own.

Fetching all 20 rows in one batch matters. Twenty calls made one after another would add their waiting times together, and the feed would feel slow even with fast parts.

Most problems come from a few very popular things, from deletion, and from privacy. Here is what the team planned for after the festival weekend.

• A viral post: one row is read millions of times. Keep hot rows in the cache, and let the CDN carry the image.
• Like counts: thousands of likes a second on one row make writers wait for each other. Split the count into several smaller counters and add them when reading.
• Deleted photos still on the CDN: long caching means copies linger. On delete, mark the row deleted and ask the CDN to remove those keys.
• Private accounts: a copied image link could leak. Serve private photos through signed links that stop working after a short time.
• Festival backlogs: the resize queue grows. Show owners their own photo at once from the phone, and add workers based on queue length.

Originals need the most care, because resized copies can always be made again from them. Haystack saved each photo size in 3 different locations.

Old photos still cost storage. Facebook's 2014 f4 paper describes moving warm data, read far less often, out of Haystack; photos moved after about three months.

f4 cut the effective replication factor, physical bytes stored divided by logical bytes, from 3.6 to 2.8 or 2.1. It still survives disk, host, rack and data centre failures.

They break in four tested places. One shard makes more than 1,024 posts in a millisecond, a clock moves backwards, or two shards post in the same millisecond. Stored in a signed 64-bit column, one that also holds negative numbers, they turn negative after about 34.8 years.

Instagram described the scheme in an April 2012 talk and a blog post, Sharding and IDs at Instagram. Each 64-bit ID holds 41 bits of time in milliseconds, 13 bits of shard ID and 10 bits of a counter.

They generated IDs inside PostgreSQL itself, with a small function in PL/pgSQL, PostgreSQL's built-in language, on each logical shard. Logical shards are many small slices, thousands of them, mapped onto fewer machines.

Here is the same layout in Python. It takes the time as an argument so the output is repeatable. The app's own epoch, its zero point, is 1 January 2024.

EPOCH_MS = 1_704_067_200_000  # 1 Jan 2024 UTC, the app's own start

def make_post_id(now_ms: int, shard_id: int, seq: int) -> int:
    post_id = (now_ms - EPOCH_MS) << 23   # 41 bits of time
    post_id |= (shard_id % 8192) << 10    # 13 bits of shard
    post_id |= seq % 1024                 # 10 bits of counter
    return post_id

def shard_of(post_id: int) -> int:
    return (post_id >> 10) & 0x1FFF

def created_ms(post_id: int) -> int:
    return (post_id >> 23) + EPOCH_MS

shard = 31341 % 2048                      # user 31341, 2,048 logical shards
now_ms = 1_792_405_800_000                # 19 Oct 2026, 10:30 UTC
pid = make_post_id(now_ms, shard, 5000)
print(pid, pid.bit_length())
print(shard_of(pid), created_ms(pid) == now_ms)

We ran it on Python 3.13.16. The first line is the ID and its length in bits; the second is the shard read back and whether the time round-trips.

741037886669436808 60
621 True

The shard is 621 because 31341 divided by 2,048 leaves 621. Instagram's blog example used 2,000 logical shards, so the same user landed on shard 1341 there.

Send these IDs to apps as strings. JavaScript numbers hold every whole number exactly only up to 2 to the power 53, and this ID is bigger. In Node 22, JSON.parse, the built-in reader for JSON text data, read it as 741037886669436800.

So the order promise is exact only to the millisecond, and only while clocks move forward. Inside one millisecond, IDs sort by shard and counter, not by arrival.

For a profile grid this is fine, because one user's posts share one shard and the order inside a millisecond does not matter. For anything that needs strict order, add a tie-breaker or a single sequence.

The top time bit is also the sign bit, the bit that marks a number as negative. PostgreSQL's bigint is signed. Instagram's blog post declares its ID column as bigint.

That means 41 bits of time would last 69.7 years only in an unsigned column. In a signed one, plan for about 34.8 years from your epoch.

Logical shards also make growth cheap. If a user's shard is the remainder of user ID divided by the number of machines, adding a machine moves almost every user. With many logical shards, you move a few whole shards and leave the rest alone.

Keep signed links short, tied to one content type and capped in size, give each upload a fresh key, and make the worker fully decode every file before trusting it. Each of these was checked below.

Check what a signed link allows

The signing itself happens on your server, with no network call. This snippet uses fake keys, so you can run it anywhere with boto3, AWS's Python library, installed. We also ran it with networking switched off.

# Signing happens locally: no network call is made. Fake keys, for the demo only.
import boto3
from urllib.parse import urlsplit, parse_qs

s3 = boto3.client("s3", region_name="ap-south-1",
                  aws_access_key_id="AKIAEXAMPLEONLY", aws_secret_access_key="example-secret")
url = s3.generate_presigned_url(
    "put_object",
    Params={"Bucket": "photos-originals", "Key": "uploads/u-7f3a/original.jpg",
            "ContentType": "image/jpeg"},
    ExpiresIn=600,
)
parts = urlsplit(url)
q = parse_qs(parts.query)
print(parts.netloc + parts.path)
print("X-Amz-Expires", q["X-Amz-Expires"][0])
print("X-Amz-SignedHeaders", q["X-Amz-SignedHeaders"][0])

On boto3 1.43.115, as of 19 October 2026, it prints the host and path, then two values from the link. X-Amz-Expires is the lifetime in seconds.

photos-originals.s3.amazonaws.com/uploads/u-7f3a/original.jpg
X-Amz-Expires 600
X-Amz-SignedHeaders content-type;host

The content type is part of the signature, so the phone must send the same Content-Type header. AWS's S3 guide lists a mismatched content type as a cause of failed uploads.

The same guide says links made with the CLI or SDKs can last up to 7 days. For a phone upload, minutes are enough.

This link signs only the content type and host, so it does not cap the file size. Without a cap, the worker's size check runs only after the file is stored.

To cap it, use a presigned POST instead: a signed upload form whose policy lists what the upload may contain. AWS's POST policy page describes a content-length-range condition with a minimum and maximum size, and boto3 put our range into the policy.

Work out how late the expiry safety net can be

AWS's lifecycle troubleshooting page gives the rule. S3 counts an object's age from its creation time in UTC, then rounds the expiry to midnight UTC the next day.

Take a story posted at 6:30 pm IST on 19 October 2026, which is 13:00 UTC. With the two-day rule, it becomes two days old at 13:00 UTC on 21 October.

It is eligible at 00:00 UTC on 22 October, which is 5:30 am IST, 59 hours after posting. Across a whole day of posting times, the gap runs from 48 to 72 hours.

Removal is asynchronous, so the real deletion can come later still. To check one object, send a HEAD request, which fetches only an object's headers. The S3 API reference documents an x-amz-expiration response header for this.

x-amz-expiration: expiry-date="Thu, 22 Oct 2026 00:00:00 GMT", rule-id="stories-safety-net"

That line is the expected header, worked out from the documented rule and the header format in the API reference. We could not run it against real S3; tests/lifecycle_date.py does the date sums.

Make the resize worker reject bad files

The worker below checks file size and pixel count, decodes the whole image, fixes the camera rotation, and writes three JPEG copies without the original's EXIF data. EXIF is hidden camera metadata, which can include GPS location.

The copies keep an RGB original's colour profile, data that tells viewers how to show its colours. Without the icc_profile argument, our test copies lost it. Small photos are never enlarged.

import io
from PIL import Image, ImageOps

WIDTHS = {"grid": 150, "feed": 640, "full": 1080}
MAX_BYTES = 20_000_000
MAX_PIXELS = 60_000_000                      # your own cap: Pillow errors only above 2x its limit

def make_sizes(original: bytes) -> dict:
    if len(original) > MAX_BYTES:
        raise ValueError("file too large")
    img = Image.open(io.BytesIO(original))   # reads the header: fails on non-images
    if img.width * img.height > MAX_PIXELS:
        raise ValueError("too many pixels")  # checked before decoding
    img.load()                               # decodes fully: fails on cut-off files
    img = ImageOps.exif_transpose(img)
    icc = img.info.get("icc_profile") if img.mode == "RGB" else None
    img = img.convert("RGB")
    out = {}
    for name, w in WIDTHS.items():
        w = min(w, img.width)                # never enlarge a small photo
        copy = img.resize((w, round(img.height * w / img.width)))
        buf = io.BytesIO()
        copy.save(buf, "JPEG", quality=80, icc_profile=icc)  # no exif, so no GPS
        out[name] = buf.getvalue()
    return out

if __name__ == "__main__":
    data = open("test_photo.jpg", "rb").read()
    print("original", round(len(data) / 1e6, 2), "MB")
    for name, b in make_sizes(data).items():
        print(name, Image.open(io.BytesIO(b)).size, round(len(b) / 1e3), "KB")

We ran it with Pillow 13.0.0 on a synthetic 12-megapixel test image with GPS data in it. Real photos compress differently, so read the sizes as an example, not a benchmark.

original 5.43 MB
grid (150, 112) 9 KB
feed (640, 480) 92 KB
full (1080, 810) 201 KB

On this image, nine grid copies came to about 78 KB, against 5.4 MB for the original. None of the three copies kept the GPS tag; tests/resize_checks.py checks that. tests/independent/worker_checks.py covers the colour profile, small photos and the pixel cap.

Pillow also guards against decompression bombs, tiny files that unpack into huge images. In Pillow 13.0.0, opening an image over 178,956,970 pixels, twice its MAX_IMAGE_PIXELS default, raises DecompressionBombError. Between 89,478,485 pixels and that limit, it only warns.

Our 97 KB PNG of 100 million pixels passed the file-size check and opened with just a warning. That is why the worker sets its own MAX_PIXELS and checks it before decoding.

Explain it as two roads, one for image bytes and one for post data, then add stories as a timing problem. Keep the feed short unless the interviewer asks about it.

1. 1

   Pin down the features

   Agree on upload, follow, home feed and 24-hour stories. Park comments, search and direct messages.

2. 2

   Size the load

   Estimate uploads, views and storage per day. Point out that views beat uploads by hundreds to one.

3. 3

   Draw the upload path

   Signed link to object storage, confirm call, queue, resize worker, then mark the post live.

4. 4

   Draw the read path

   CDN for image bytes, and a cache in front of a sharded metadata database for post rows.

5. 5

   Explain IDs and sharding

   Time-sorted 64-bit IDs made on each shard, many logical shards on fewer machines, the 1,024-per-millisecond limit and the signed-column limit.

6. 6

   Add stories

   Hide on read after 24 hours, archive or delete files later, and use a storage expiry rule only as a backup.

7. 7

   Close with failures

   Cover viral posts, deletes on the CDN, private photos and keeping originals safe.