YouTube System Design: Upload, Transcode and Stream

Uploads fail because one long request must survive a shaky network, and playback stalls because every viewer gets the same heavy file. A young video site often hits both at once, and fixing them is what YouTube system design is about.

Picture a common setup. A regional video app lets creators post comedy sketches, folk songs and travel vlogs from their phones. The setup is made up, but the problems are real.

The first version was the naive one, and it was wrong in a way that only shows at scale. The phone sent the whole file in one HTTP request to one server. That server saved it and played the same file to everyone.

Now the app gets about 2,000 uploads a day and 3 lakh (300,000) daily viewers. Three complaints keep coming back.

• A creator in a hill village uploads a 1.2 GB video. The connection drops at 80 percent, and the upload starts again from zero.
• A viewer on patchy 4G presses play. The single 1080p file needs more speed than her network has, so the video stops every few seconds.
• A song goes viral on a Sunday evening. Everyone hits the same server, playback slows for all viewers, and the bandwidth bill jumps (made up: Rs 4.2 lakh, about Rs 420,000, that month).

Bigger platforms avoid these problems by splitting the work into three separate jobs. The rest of this guide rebuilds the app the same way, one job at a time.

YouTube system design is the design of a service that accepts large video uploads, converts each video into many sizes, and streams it smoothly to huge numbers of viewers.

The design splits into three paths that rarely talk to each other directly. The upload path takes files in, the processing path transcodes them, and the watch path streams them out. Files live in object storage, a service that stores huge files cheaply under a name.

The words you need first

Codec. A codec is a method for compressing video, such as H.264 or AV1. Raw camera video is huge. The codec shrinks it, and one big trick is storing only what changes between frames.

Keyframe. A keyframe is a complete picture. The frames after it store only changes, so a player or decoder can only start cleanly at a keyframe.

Bitrate. Bitrate is how many bits of video play each second, written in kbps or Mbps (thousands or millions of bits per second). A higher bitrate usually means a sharper picture and a bigger file.

Rendition and ladder. Transcoding means decoding a video and encoding it again with different settings. Each result is a rendition: one resolution at one bitrate. The full set is called the encoding ladder, or bitrate ladder.

Segment and CDN. A segment is a short piece of one rendition, a few seconds long. A CDN, or content delivery network, is a set of servers in many cities that keep copies of popular files close to viewers.

What the system must do

• Creators upload videos of up to 4 GB from phones, and an upload survives dropped connections.
• Each video becomes watchable within minutes, in several qualities.
• Playback starts in about two seconds and rarely shows the spinning wheel.
• Viewers see video details, view counts and comments.
• Watching keeps working even if uploads or processing are down.

The rough maths

Do the maths before drawing boxes. A typical upload is 10 minutes long and 1.2 GB raw, which is 16 Mbps. The five renditions in the ladder below add 780 MB of video. So 2,000 uploads add about 4 TB of storage a day.

Watching is much bigger. Say 3 lakh viewers watch 40 minutes a day at an average of 1.4 Mbps. That is about 126 TB sent out daily, roughly 52 times the 2.4 TB of raw uploads coming in.

You can check these with a calculator, using decimal units (1 GB is 1,000,000,000 bytes). For example, 300,000 x 2,400 seconds x 1.4 million bits, divided by 8, is 126 x 10^12 bytes, or 126 TB.

Here is the full design. Follow the creator's phone at the top through storage and the job queue. Then notice that the viewer's player gets video only from the CDN, and video details from a small API, never from the upload service.

A resumable upload sends the file in parts and lets the client ask the server how much it already has. So a dropped connection loses only the part in flight, and the creator continues from there.

The naive version sent all 1.2 GB in one HTTP request. When the phone switched from Wi-Fi to mobile data, the request died, and the server threw everything away.

The IETF, the group that writes internet standards, is drafting a standard for this called Resumable Uploads for HTTP. As of 19 October 2026 it is draft 14, dated 18 October 2026, and not yet an RFC, the IETF's published document series. Its design is worth copying even before it is final.

• The client starts an upload with an ordinary request carrying an Upload-Complete header. A server that supports resuming can reply early with a 104 status and the upload's own address.
• The client sends parts to that address with PATCH, the HTTP method for changing part of a resource. Each part carries an Upload-Offset header: the number of bytes sent before it.
• After a drop, the client sends a HEAD request, which asks for headers only. The server replies with the offset it has safely stored.
• The client continues from that offset. Its last request sets Upload-Complete to true, and the server can announce limits, such as how long the upload address lives, in an Upload-Limit header.

The draft can still change before it becomes an RFC, so pin the draft number you build against. The diagram follows the hill-village creator through one dropped connection. Each box is one message.

Here is the client side in Python, with a fake upload service that drops 40 percent of requests after some bytes have landed. Run it with Python 3.13; the seed makes the drops repeatable.

import os, random, tempfile

PART_SIZE = 8 * 1024 * 1024                  # 8 MiB per request

class FlakyUploadService:                    # stands in for the real upload service
    def __init__(self):
        self.stored, self.complete = bytearray(), False
    def head(self):                          # HEAD: how many bytes are safely stored?
        return len(self.stored)
    def patch(self, offset, data, last):     # PATCH: append one part at this offset
        if offset != len(self.stored):
            raise ValueError("409 Conflict: wrong offset")
        if random.random() < 0.4:            # the signal drops mid-request...
            self.stored += data[:random.randrange(len(data) + 1)]  # ...after some bytes landed
            raise ConnectionError("signal lost")
        self.stored += data
        self.complete = last
        return len(self.stored)

def upload_video(path, service):
    offset = service.head()                  # ask, never assume
    sent = drops = 0
    with open(path, "rb") as f:
        while not service.complete:
            f.seek(offset)                   # skip bytes the server already has
            data = f.read(PART_SIZE)
            last = f.tell() == os.path.getsize(path)
            try:
                offset = service.patch(offset, data, last)
            except ConnectionError:
                drops += 1
                offset = service.head()      # the part may have partly landed
            sent += len(data)
    return sent, drops

random.seed(7)
path = os.path.join(tempfile.mkdtemp(), "video.mp4")
video = os.urandom(50 * 1024 * 1024)         # a 50 MiB stand-in for a video
open(path, "wb").write(video)
service = FlakyUploadService()
sent, drops = upload_video(path, service)
print("dropped requests:", drops)
print("MiB re-sent:", round((sent - len(video)) / 2**20, 1), "of at most", drops * 8)
print("server copy matches:", bytes(service.stored) == video)

Expected output on Python 3.13: dropped requests: 7, then MiB re-sent: 45.6 of at most 56, then server copy matches: True. MiB means mebibytes, 1,048,576 bytes each.

What just happened? Seven requests died, yet the server's copy matches the file byte for byte. Each retry first asked the server for its offset, because a dropped part may have partly landed. In total, it re-sent no more than one 8 MiB part per drop. A property test over 270 runs, including empty files and exact multiples of 8 MiB, held the same bound.

Part size has limits there too. S3 multipart parts must be 5 MiB to 5 GiB, except the last, with at most 10,000 parts. Cloud Storage wants chunks in multiples of 256 KiB. An 8 MiB part fits both. At that fixed size, 10,000 parts cap one S3 upload at about 78 GiB.

After upload, the video goes through a processing pipeline: check it, split it, transcode it into several renditions, package the results, then mark it ready. Each step is a job on a queue, run by workers.

Think of a rice mill. Paddy comes in by the truckload, gets husked and graded, then is packed in 1 kg, 5 kg and 25 kg bags, because different shops need different sizes.

Where the picture breaks: a mill fills each bag from the same rice, but each rendition is a fresh encode. A 1080p rendition is not five 240p renditions stuck together, and you cannot make a sharp 1080p from 240p.

The FFmpeg documentation, for the widely used open-source video tool, defines transcoding as decoding a stream and then encoding it again. Its pipeline reads the file, decodes it into frames, filters them, encodes them and writes them out. The app's workers run that pipeline.

Here is the app's ladder. The top rendition needs 12.5 times the bitrate of the bottom one.

Splitting the work into chunks

Encoding a 10-minute video into five renditions on one machine is slow. Say one worker needs about 25 minutes (made up). That is too long for a creator waiting to share a link.

So the pipeline cuts the video into chunks of 4 seconds and encodes them in parallel. A 10-minute video gives 150 chunks, and five renditions make 750 small jobs. Spread over 100 workers, that is about 15 seconds of encoding each, before queue waits and packaging.

The diagram shows one upload moving through the pipeline. Watch the middle step, where one job becomes hundreds.

Codec choice adds one more trade-off. MDN Web Docs, Mozilla's developer reference, calls H.264 broadly compatible, so every video gets it. It credits AV1 with up to 50 percent higher compression than H.264. As of 19 October 2026, it says Safari plays AV1 only on devices with a hardware decoder.

Encoding time depends heavily on the encoder and its speed preset. On a 10-second 360p test clip (2 cores, quality not matched), H.264 with x264's medium preset took about 3 seconds. AV1 with SVT-AV1, an open-source AV1 encoder, took 2.5 to 4 seconds at its fast preset 12 and about 30 at preset 4.

Adaptive bitrate streaming lets the player switch renditions every few seconds, based on network speed and how much video it has already downloaded. Each segment can come from a different rendition, so quality changes without stopping playback.

A widely used format is HLS, short for HTTP Live Streaming, described in RFC 8216, an informational IETF document from August 2017. MPEG-DASH (Dynamic Adaptive Streaming over HTTP) is an international standard built on the same idea. Both run over plain HTTP, so any web server or CDN can deliver them.

HLS uses small text files called playlists. A master playlist lists every rendition with its peak bitrate and resolution. Each rendition then has a media playlist, which lists its segments in order.

The player downloads the master playlist first, picks a rendition and starts fetching segments. RFC 8216 says clients should switch between renditions to adapt to network conditions. It leaves the switching rule to the player.

That rule matters a lot. The buffer is the video already downloaded but not yet played. Rebuffering means playback stops while the player waits for more data: the spinning wheel viewers hate.

Here is what happens when a viewer presses play.

1. 1

   Get the video details

   The app asks the API for the title, status and master playlist address. These come from the metadata database, usually through a cache.

2. 2

   Load the master playlist

   The player reads the list of renditions and their bitrates. It is a tiny text file, served by the CDN like any segment.

3. 3

   Start low

   The player picks a modest rendition, such as 360p, so the first frame appears fast.

4. 4

   Fill the buffer

   It downloads the first segments from the nearest CDN server and starts playing once it has enough.

5. 5

   Adjust on every segment

   As the buffer grows, it steps up to 720p or 1080p. When the network dips, it steps down before the buffer runs dry.

It serves almost every segment from a CDN, so viewers download from a server in their own region instead of from your storage. Your origin, the storage holding the master copy, only sees cache misses.

Plan for uneven traffic: a few videos get most of the views, while a long tail of older videos gets a few each. Check the shape in your own view logs, because it decides where each segment should live.

• Hot videos are new or viral. Their segments stay cached on CDN servers in many cities.
• Warm videos are watched now and then. The CDN fetches them from origin on the first request and keeps them for a while.
• Cold videos are rarely watched. Keep them in a cheaper storage class, and consider keeping only some renditions.

Back to the viral Sunday song. With a CDN, the first viewer in a city pulls each segment from origin once. The next ten thousand viewers there get the edge server's copy, so origin load barely moves.

Now check the peak. At 9 p.m., about 40,000 people watch at once at about 1.4 Mbps each. That is 56 Gbps leaving the system, which a CDN spreads across many cities and networks.

Segments never change once written, so the CDN can cache them for a long time. If you re-encode a video, the new segments get new names, so no viewer gets a mix of old and new.

A worker must start decoding at a keyframe, because the frames after it store only changes. Its chunk boundary need not be one, though. The worker can decode from the keyframe before its start and throw the extra frames away. The encoder then opens each output chunk with a fresh keyframe. Only stream copy, which cannot add keyframes, is stuck with the keyframes the source already has.

Segments follow a softer rule. RFC 8216 says each video segment should carry enough to start a decoder, and an H.264 segment should contain an IDR frame, a keyframe that resets decoding. That is what lets a player jump into any segment when it switches rendition.

Pause and predict. Take a 30-second clip at 30 frames per second, encoded with ffmpeg 6.1.1 and x264, the open-source H.264 encoder, at default settings. Ask ffmpeg to cut 4-second HLS segments with stream copy (-c copy, which skips decoding and encoding). How long are the segments?

They came out 8.33, 8.33, 8.33 and 5 seconds long on ffmpeg 6.1.1. The clip had a keyframe every 250 frames, which is 8.33 seconds at 30 frames per second. Stream copy cannot add keyframes, and ffmpeg's docs say the segment is cut on the next keyframe after the time has passed.

Re-encoding with a keyframe forced every 4 seconds fixed it: seven 4-second segments and a 2-second tail. In ffmpeg that option is -force_key_frames with the expression gte(t,n_forced*4).

To check your own file, list its keyframe times. In the command, source.mp4 is your video.

ffprobe -v error -select_streams v -skip_frame nokey \
  -show_entries frame=pts_time -of csv=p=0 source.mp4

Expected output for the test clip: four lines reading 0.000000, 8.333333, 16.666667 and 25.000000 seconds. In our run, ffprobe printed a stray comma after the first. If the gaps are longer than your segment length, stream copy gives you longer segments. The playlist's target duration, its stated maximum segment length, grows with them.

Parallel encoding shows the other side. In our test, each worker took a 4-second window with -ss, ffmpeg's start-time option, placed before the input. Only the first window started on a source keyframe. ffmpeg seeks to the keyframe before the start, then decodes and discards frames up to it. The worker's first decoded frame matched the frame at 4.0 seconds in a full decode exactly. Every encoded chunk began with a keyframe, and the joined result kept all 900 frames.

Planning the jobs

Here is the job planner. LADDER maps the short side of each rendition, in pixels, to its bitrate in kbps. The set done_ids holds job ids already finished, read from the metadata database.

import math

LADDER = {240: 400, 360: 800, 480: 1400, 720: 2800, 1080: 5000}  # short side -> kbps
CHUNK_SECONDS = 4

def plan_jobs(video_id, duration_s, short_side):
    jobs = []
    for lines, kbps in LADDER.items():
        if lines > short_side:               # never upscale
            continue
        for n in range(math.ceil(duration_s / CHUNK_SECONDS)):
            jobs.append({"id": f"{video_id}:{lines}p:{n}",   # same work, same id
                         "start_s": n * CHUNK_SECONDS, "lines": lines, "kbps": kbps})
    return jobs

def enqueue_missing(queue, jobs, done_ids):
    for job in jobs:
        if job["id"] not in done_ids:        # a rerun skips finished work
            queue.append(job)

jobs = plan_jobs("v42", 600, 1080)
print("10-minute 1080p upload:", len(jobs), "jobs")
print("10-minute 720p upload:", len(plan_jobs("v43", 600, 720)), "jobs")
print("10-minute portrait phone video, 1080 x 1920:", len(plan_jobs("v44", 600, 1080)), "jobs")
queue, done = [], {j["id"] for j in jobs[:500]}  # a crash after 500 jobs finished
enqueue_missing(queue, jobs, done)
print("rerun after a crash queues:", len(queue), "jobs, first:", queue[0]["id"])

Expected output: 750 jobs for a 10-minute 1080p upload, 600 for a 720p one and 750 for a portrait 1080 x 1920 phone video. After the crash, 250 jobs are queued again, starting at v42:720p:50.

What just happened? The job id is built from the video, rendition and chunk, so the same work always gets the same id. A rerun skips the 500 finished jobs. Comparing the short side stops a 720p upload being stretched into a blurry 1080p rendition.

One well-tested answer is to choose mainly from the buffer level, and look at how fast chunks arrive only when the buffer says little, such as at startup. Estimating network speed alone can cause needless rebuffering.

That answer comes from a 2014 paper by Huang, Johari, McKeown, Trunnell and Watson. It argues for dropping capacity estimates and controlling the buffer directly. Its buffer-based rules ran inside a large commercial service's video client, in experiments with millions of users.

The paper reports rebuffer rates down by 20 percent while holding the video rate constant. It used 4-second chunks and a 240-second buffer, so its numbers fit that setup, not every player.

Here is a simple buffer rule this app's player could start with. Begin at the top and follow the buffer level to the next request.

The same paper found a pure buffer rule too cautious at startup, because an almost empty buffer says little. Its improved version watched how much the buffer grew during each chunk download, and stepped up early when chunks arrived fast.

What the bandwidth number in a playlist really means

RFC 8216 says a rendition's BANDWIDTH value represents its peak segment bit rate. Peak means the highest rate over any run of segments lasting 0.5 to 1.5 times the target duration. When all segments already exist, it must equal the real peak, summed over renditions that play together.

Players read this number to decide which rendition their connection can afford, so an understated value hurts.

ffmpeg 6.1.1 does not measure it. Its HLS writer adds the bit rates recorded on the video and audio streams, then adds 10 percent (libavformat/hlsenc.c, lines 1484 to 1489).

Those recorded rates are your targets when you set -b:v and -b:a, ffmpeg's video and audio bitrate options. With stream copy, they are the source file's recorded averages. Without -b:v, x264 runs in its default quality mode and ffmpeg records no video rate. Our test then declared 75,900 bit/s, the audio alone plus 10 percent.

In our 3-rendition test, the 240p rendition was declared at 545,600 bit/s, which is 400,000 plus 96,000, plus 10 percent. Its largest real segment ran at 567,760 bit/s. All three renditions peaked 3 to 6 percent above their declared value, on a synthetic clip with maxrate, ffmpeg's bitrate cap, set equal to the target.

The muxer documentation describes neither rule, in 6.1.1 or in 8.1.3. ffmpeg 7.1 changed the code. When a finished run writes its final master playlist, each BANDWIDTH is the rendition's largest measured segment rate. It also adds AVERAGE-BANDWIDTH, the average rate over the whole rendition. Playlists written before the end, as in live streaming, still use the recorded rates plus 10 percent. We read this in the 7.1 and 8.1.3 source but could not run those versions, so check your own output and pin your ffmpeg version.

To check yours, divide each segment file's size in bits by its EXTINF duration, the seconds listed above it in the media playlist. Compare the largest result with BANDWIDTH. If it is higher, a player may pick a rendition its network cannot sustain.

Video details live in a metadata database, view counts in a separate counting pipeline, and comments in their own store. Keeping them apart stops a busy comment section from slowing down playback.

The metadata database holds one row per video: owner, title, status, duration and the master playlist address. It is small next to the videos. But it is read on every page view, so a cache sits in front of it.

When one database is not enough, you shard it: split one big table across many databases by a key, such as the video id.

YouTube hit this wall. Vitess was created in 2010 to solve the scaling problems YouTube faced with MySQL, a popular open-source database. It puts a proxy, a middle server, between the app and many MySQL shards that routes each query to the right one. It graduated as a CNCF (Cloud Native Computing Foundation) project in November 2019.

View counts need a different approach. One database update per view would swamp the database during a viral spike. Instead, each view becomes an event on a queue, and a counter service adds them up every few seconds.

The count on the page runs slightly late, and nobody minds seeing 10,214 views instead of 10,240. Creator payouts still need exact totals, so a daily job counts every view event.

The usual breakages are stuck or repeated jobs, half-processed videos, bad uploads and storage that grows forever. Each one needs a guard designed in from the start, not patched in later.

• Stuck jobs. A worker dies mid-encode and the video sits in processing forever. Guard: every job has a time limit, and the queue hands timed-out jobs to another worker.
• Repeated work. A retried job encodes the same chunk twice. Guard: the fixed job ids from the planner above.
• Bad uploads. A file claims to be video but is not, or is corrupt halfway. Guard: check it first and reject it with a clear reason.
• Half-ready videos. Some renditions finish and others fail. Guard: mark a video ready only when every rendition in its master playlist exists.
• Storage that only grows. Abandoned uploads leave parts behind. Guard: delete unfinished upload sessions after they expire.

Walk through the three paths in order, with numbers, and spend most of your time on processing and playback. That is where video differs from other systems.

1. 1

   Agree on scope

   Ask about upload size, daily uploads, daily viewers and whether live streaming is included. Leave live out unless the interviewer wants it.

2. 2

   Estimate

   Work out storage added per day and bandwidth sent out. Show that watching moves far more data than uploading.

3. 3

   Draw the three paths

   Upload, processing and watching, joined only by object storage, a job queue and the metadata database.

4. 4

   Go deep on processing

   Explain the encoding ladder, chunks that start on keyframes, parallel jobs and job ids that make retries safe.

5. 5

   Go deep on playback

   Explain playlists, segments, a buffer-based quality rule and why the player starts at low quality.

6. 6

   Scale the reads

   Add a CDN with an origin shield, hot and cold storage, a cached and sharded metadata database, and view counts through a queue.

7. 7

   Name the failures

   Stuck jobs, repeated work, half-ready videos and viral spikes, with one guard each.