Design Video Conferencing Like Zoom: SFU vs MCU and WebRTC

The video freezes, because every phone now sends and receives more video than its network can carry. The fix is not a faster phone. It is a different shape for the call, and that shape is the core of video conferencing system design.

Picture a common setup. A small engineering team runs a telehealth service that lets doctors at a city clinic in Visakhapatnam see patients in coastal villages over video. The setup is made up, but the problem is real.

The naive first version connected the doctor and patient directly, phone to phone. For two people it worked well. Then the clinic started group consults: a doctor, a patient, a family member in Hyderabad, an interpreter and sometimes a heart specialist.

With five people, every phone had to send its camera four times, once to each other person. A patient on village 4G had about 3 Mbps (megabits per second) of upload. Four HD (high-definition) copies needed about 6 Mbps (made up). The patient's video froze.

The rest of this guide follows the team as it rebuilds the service one decision at a time, with the same five-person consult as its test case.

Video conferencing system design is planning how live audio and video travel between many people with low delay, good quality and privacy. It covers the app, the servers that set up calls, and the servers that carry media.

Live video is different from a video site. On a video site, the player can buffer (load a few seconds ahead) and nobody minds. In a call, a two-second delay makes people talk over each other. So a call keeps only a small buffer, and data that arrives too late to play is skipped rather than waited for.

Before drawing any boxes, the team wrote down what the service must do:

• Start a call in a few seconds from a link sent by SMS, with no app install for family members.
• Support rooms of 2 to 8 people, with about 1,500 rooms at the busiest hour (made up).
• Keep delay low enough for a natural talk, with a target of 300 ms (milliseconds) one way (made up).
• Keep working on weak village networks, even if video drops to a small picture or audio only.
• Encrypt every call, because consults carry private health details, and record only when the patient agrees.

The first four are about the network. The last is about trust. Most of the design comes down to one question: where does the video go between the sender and the receivers?

WebRTC connects two people in three moves: the apps swap call details through signaling, find a working network path with ICE, then send encrypted media over that path. Modern browsers ship WebRTC built in, and native apps add a WebRTC library.

WebRTC (Web Real-Time Communication) is a set of browser APIs and network protocols for live audio, video and data. Its main API object is RTCPeerConnection, which holds one connection to another endpoint.

Signaling: swapping call details

Before any video flows, the two apps must agree on details. These include which codecs (ways of compressing video) they support and which network addresses might work. The details travel as an offer and an answer written in SDP, a plain text format for describing sessions.

The W3C WebRTC spec, a W3C Recommendation whose latest version is dated 13 March 2025 as of 19 October 2026, does not define how they travel. It says they go by unspecified means. So you build signaling yourself. The team uses a WebSocket (a connection that stays open so either side can send at any time) to its own signaling server.

ICE, STUN and TURN: finding a path

Many phones and laptops sit behind NAT, a router feature that shares one public address among many devices. NAT hides the device's real address, so the other side cannot simply connect to it. ICE (Interactive Connectivity Establishment) is the method for finding a path anyway. It is defined in RFC 8445; an RFC is a numbered internet standards document.

• Host candidate: an address on the device's own network card. It works when both people sit on the same network.
• Server reflexive candidate: the public address the NAT gives the device. A STUN server tells the device what that address is.
• Relayed candidate: an address on a TURN server, which forwards every packet for you. It works on almost any network, but you pay for the bandwidth.

RFC 8445 also defines a fourth kind, the peer reflexive candidate, which an app discovers during the checks themselves. Each side sends its candidates through signaling and tests pairs with small STUN requests. One side, the controlling agent, then nominates a working pair.

ICE prefers direct routes. RFC 8445 recommends a type preference of 126 for host candidates and 0 for relayed ones, so TURN is the last resort. Follow the boxes below from address gathering to the media flowing.

RFC 8656, the TURN standard, notes that some firewalls block UDP entirely. UDP is quick delivery with no built-in resending; TCP is reliable, in-order delivery. TURN's default ports are 3478 for UDP and TCP, and 5349 for TURN over TLS (the encryption behind HTTPS). You can also run TURN over TLS on port 443, the HTTPS port, so a firewall that allows web traffic may let it through.

In the team's logs, about 15 percent of calls end up relayed (made up). Your own rate decides your TURN bill.

Mesh breaks down because each person must upload a separate copy of their video to every other person. Upload is often the slower direction of a home or mobile connection, so it runs out first.

In a mesh, every pair of people has its own RTCPeerConnection. With n people, each one sends n minus 1 streams and receives n minus 1 streams. The phone also compresses its camera picture once per copy, which drains the battery.

Pause and predict: with a media server in the middle, how many video streams does each person upload in the five-person consult? The short Python script below counts them. It assumes 1,500 kbps (kilobits per second) for one HD stream, a made-up round number.

HD_KBPS = 1500  # one HD stream, a made-up round number
UPLOAD_KBPS = {
    "mesh": HD_KBPS,              # each copy is a full HD stream
    "sfu": 1500 + 500 + 150,      # one upload: high, medium and low encodings
    "mcu": HD_KBPS,               # one HD stream to the mixer
}

def streams_per_person(n, topology):
    """Streams one person sends and receives in an n-person call."""
    others = n - 1
    if topology == "mesh":
        return others, others  # a copy to and from every other person
    if topology == "sfu":
        return 1, others       # one upload to the SFU, one stream per other person
    if topology == "mcu":
        return 1, 1            # send once, receive one mixed picture
    raise ValueError(topology)

for t in ("mesh", "sfu", "mcu"):
    up, down = streams_per_person(5, t)
    print(f"{t}: sends {up}, receives {down}, uploads {up * UPLOAD_KBPS[t]} kbps")

Run it with python3 and you should see these three lines:

mesh: sends 4, receives 4, uploads 6000 kbps
sfu: sends 1, receives 4, uploads 2150 kbps
mcu: sends 1, receives 1, uploads 1500 kbps

What just happened? Mesh uploads grow with the room: 6,000 kbps for five people, double the patient's 3 Mbps. With an SFU, the patient sends one upload to the server carrying three separate encodings, 2,150 kbps in total, whatever the room size. An MCU also cuts downloads to one stream, but the server pays for that in processing.

An SFU (Selective Forwarding Unit) is a media server that receives each person's stream and forwards copies to the others, without decoding or mixing the video. It picks which streams, and which quality, each receiver gets.

Picture a Carnatic kutcheri (a classical concert) in a sabha hall. The singer, violinist and mridangam player each have a stage monitor, a small speaker that faces the performer. Each artist hears every other musician as a separate feed and sets their own levels.

The violinist turns the singer up to follow the melody closely. The mridangam player turns the violin down. Nobody blends the music for them. That is an SFU: separate feeds, chosen per listener, with the combining done at each end.

Where the picture breaks: stage monitors carry sound instantly across a room. An SFU sends compressed packets over a network, and every receiver must decode each stream it gets, which costs CPU and battery on the receiver's device.

RFC 7667 from the IETF, the body that writes internet standards, describes this box. It covers RTP, the standard packet format for live media, and calls the box a Selective Forwarding Middlebox. It works without signal processing, and the RFC says SFMs have extremely low delay compared with MCUs that transcode (decode and re-encode) media.

The cost moves to the receivers. RFC 7667 says each receiver needs as many decoders as streams it receives, or must switch a smaller set of decoders between streams.

Simulcast: several qualities from one sender

Simulcast means one sender encodes the same camera at several qualities at once, often high, medium and low. RFC 8853 defines how to signal this in SDP and RTP, with each encoding sent as its own RTP stream. This guide calls those encodings layers for short. The SFU then forwards one layer to each receiver, based on their bandwidth and tile size.

In the diagram, follow the patient's three layers into the SFU. Notice that each receiver gets only one layer, chosen for them.

RFC 8853 says the choice of which simulcast stream to forward is application specific. The team's rule is simple: split the receiver's measured download across the tiles on screen, keep some spare, then pick the best layer that fits.

LAYERS = [("high", 1500), ("medium", 500), ("low", 150)]  # kbps, best first

def pick_layer(receiver_kbps, tiles):
    """Choose the simulcast layer for each tile one receiver shows."""
    # receiver_kbps is the download left after audio has its share
    share = receiver_kbps * 0.8 / max(tiles, 1)  # keep 20% spare
    for name, kbps in LAYERS:
        if kbps <= share:
            return name
    return "audio-only"

print("doctor:", pick_layer(2500, 1))      # clinic broadband, patient full screen
print("daughter:", pick_layer(2500, 4))    # home Wi-Fi, four tiles
print("specialist:", pick_layer(800, 4))   # weak hotel Wi-Fi, four tiles
print("weakest:", pick_layer(500, 4))      # below even the low layer

Run it and you should see the same choices as the diagram, plus one receiver that falls back to sound only:

doctor: high
daughter: medium
specialist: low
weakest: audio-only

The share line matters most. It keeps 20 percent spare, so a small dip in bandwidth does not freeze the picture. The weakest receiver gets 100 kbps per tile, below even the 150 kbps low layer, so it gets audio only.

You can refine this rule. Give the active speaker a higher layer, and stop forwarding layers nobody is watching. When the app is minimised, it can ask the SFU to stop sending it video.

An MCU (Multipoint Control Unit) is a media server that decodes every incoming stream, mixes them into one picture and one sound, then re-encodes that mix for each receiver. Each person receives just one stream.

Back at the kutcheri, an MCU is the front-of-house mix. The person at the mixing desk blends all the instruments into one sound for the hall speakers. Everyone hears the same blend, and nobody can turn up just the violin.

RFC 7667 names three problems with this mixer. It consumes large amounts of media processing. Decoding and re-encoding lowers quality, like photocopying a photocopy. And the mixing adds delay, which the RFC says can be substantial.

The upside is that receivers do almost no work. An MCU still makes sense in a few places:

• Very weak devices, such as an old hospital room display that can decode only one stream.
• Phone dial-in, where a patient calls a normal number and needs one mixed audio stream.
• Recordings and broadcasts, where you want one finished video file in a fixed layout.

You can also combine the two ideas: forward video like an SFU and mix only audio, which needs far less computing than video. The team does exactly that for patients who join by phone call from villages without data.

Choose an SFU for most group calls, because it needs far less server computing, adds less delay and lets each person choose their own layout. Add MCU-style mixing only for the special cases above.

The team keeps a direct connection for one-to-one calls, where no media server is needed unless TURN has to relay. Recorded one-to-one consults go through the SFU instead, so the recorder can join. Rooms with three or more people go through an SFU. A small audio mixer serves phone dial-in.

A real offer is a block of SDP text that names the media, the network candidates and a fingerprint of a DTLS certificate. DTLS is TLS adapted for UDP, and WebRTC uses its handshake to set up the media keys. The quickest way to see one is to run two peers in one process with aiortc, an open-source Python WebRTC library.

The script below was run on aiortc 1.15.0 with Python 3.13. The doctor sends a test video track, the patient answers, and the script prints a few lines from each description, then waits for the first frame. It passes no STUN or TURN servers, so it stays on one machine; a real app lists its own servers there.

import asyncio
from aiortc import RTCConfiguration, RTCPeerConnection
from aiortc.mediastreams import VideoStreamTrack


def show(label, sdp):
    print(label)
    for line in sdp.splitlines():
        if line.startswith("m=video"):
            print("  media line, profile:", line.split()[2])
        elif line.startswith("a=candidate"):
            print("  candidate type:", line.split("typ ")[1].split()[0])
        elif line.startswith("a=fingerprint:sha-256"):
            print("  DTLS fingerprint:", line.split()[1][:17] + "...")
        elif line.startswith("a=setup"):
            print("  DTLS role:", line.split(":")[1])


async def main():
    local_only = RTCConfiguration(iceServers=[])  # no STUN or TURN: host candidates only
    doctor, patient = RTCPeerConnection(local_only), RTCPeerConnection(local_only)
    doctor.addTrack(VideoStreamTrack())  # a test picture stands in for a camera
    got = asyncio.get_running_loop().create_future()

    @patient.on("track")
    def on_track(track):
        async def first_frame():
            frame = await track.recv()
            got.set_result((frame.width, frame.height))
        asyncio.ensure_future(first_frame())

    # Signaling: in a real app these two descriptions travel over a WebSocket.
    await doctor.setLocalDescription(await doctor.createOffer())
    show("OFFER (doctor)", doctor.localDescription.sdp)
    await patient.setRemoteDescription(doctor.localDescription)
    await patient.setLocalDescription(await patient.createAnswer())
    show("ANSWER (patient)", patient.localDescription.sdp)
    await doctor.setRemoteDescription(patient.localDescription)

    w, h = await asyncio.wait_for(got, 20)
    print("ICE:", doctor.iceConnectionState, "| connection:", doctor.connectionState)
    print("first video frame at the patient:", w, "x", h, "pixels")
    await doctor.close()
    await patient.close()


asyncio.run(main())

Install it with pip install aiortc==1.15.0, then run the script. Your fingerprints will differ on every run, and a machine with several network cards prints several host candidates. The rest should match:

OFFER (doctor)
  media line, profile: UDP/TLS/RTP/SAVPF
  candidate type: host
  DTLS fingerprint: 23:86:CE:1C:24:3D... (differs per run)
  DTLS role: actpass
ANSWER (patient)
  media line, profile: UDP/TLS/RTP/SAVPF
  candidate type: host
  DTLS fingerprint: F4:FB:DC:1B:F3:33... (differs per run)
  DTLS role: active
ICE: completed | connection: connected
first video frame at the patient: 640 x 480 pixels

What just happened? The profile UDP/TLS/RTP/SAVPF means secure RTP keyed through DTLS over UDP, as defined in RFC 5764. Each side puts a hash of its DTLS certificate in its description, and RFC 8842 says the certificate in the handshake must match it.

The DTLS role lines decide who starts the handshake. RFC 8842 says the offerer must send actpass, meaning either role. Here the answerer chose active, so it sends the first DTLS message. With no STUN or TURN servers given, ICE could only use host candidates, which is enough on one machine. aiortc reports the finished ICE state as completed.

How to check whether a browser call is relayed

In a browser, call getStats() on the RTCPeerConnection and find the report of type transport. Its selectedCandidatePairId points to the candidate pair in use. That pair report has two ids: localCandidateId for your own candidate and remoteCandidateId for the other side's.

Look up both candidates. Each has a candidateType of host, srflx, prflx or relay. If either is relay, the call goes through a TURN server, yours or the other side's. The same pair report gives currentRoundTripTime in seconds and availableOutgoingBitrate in bits per second, an estimate of how much this side can send right now.

MDN, Mozilla's web developer reference, marks selectedCandidatePairId as working in the latest major browsers only since July 2026, as of 19 October 2026. Older browsers may lack it, and Firefox also offers a non-standard selected flag on candidate pairs.

You scale by keeping each room on one SFU, adding SFUs as rooms grow, and linking SFUs across regions when people in one room are far apart. The signaling service tracks which room lives where.

An SFU is stateful. It holds live connections, per-hop encryption keys and layer choices for every person in its rooms. So you cannot spread one room's packets across random servers, as you would with web requests.

1. 1

   Pick an SFU when the room opens

   The signaling service picks an SFU near the first person, with spare capacity. It saves the choice in a room directory, a fast key-value store (a lookup table from room id to SFU).

2. 2

   Send everyone to the same SFU

   Each new person asks signaling where the room lives and connects to that SFU. All of the room's media meets in one place.

3. 3

   Count capacity in bandwidth, not rooms

   A ten-person room costs far more than two small ones. The team caps each SFU at 70 percent of its network card (made up).

4. 4

   Cascade for faraway guests

   The daughter in Hyderabad connects to a nearby SFU, which links to the Visakhapatnam one. Each stream then crosses the long route once, however many guests sit in Hyderabad.

5. 5

   Drain before you remove a server

   When load drops, mark an SFU as draining. It takes no new rooms, and shuts down only after its last call ends.

Autoscaling (adding or removing servers as load changes) works here, but with care. Scale on network use and packet loss (data that never arrives), not CPU alone. Never remove a server that still has live calls, or a consult cuts off mid-sentence.

Here is the full design. Start at the apps on the left and notice two separate paths: call details go to signaling, while media goes to the SFUs.

The recording worker joins the room like a hidden participant, receives streams from the SFU, and mixes them into one file. That is MCU-style mixing, used only where one finished video is needed.

WebRTC encrypts every media stream on every hop by rule, and you add end-to-end encryption if even your own servers must not see the video. For telehealth, you also control who joins and who can record.

RFC 8827, the WebRTC security architecture, says all media must be secured with SRTP (Secure Real-time Transport Protocol) and must not be sent as plain RTP. Keys come from the DTLS handshake you saw in the offer. Calls prefer UDP because TCP delivers data strictly in order, so one lost packet would hold up every packet behind it.

A TURN server only relays packets. In a direct call through TURN, the DTLS session runs between the two apps, so the relay passes SRTP packets it has no keys for. It still sees addresses and timing.

An SFU is different. The encryption is hop by hop: phone to SFU, then SFU to phone. The SFU holds the keys for each hop, so it decrypts packets as they arrive and could, in principle, read the media.

Signaling is a trust point too: RFC 8827 warns that, even if HTTPS is used, the signaling server can potentially mount a man-in-the-middle attack. In that attack, a middle party reads or changes messages, for example by swapping the fingerprints you saw in the offer. Serve signaling over WSS (WebSocket over TLS), which stops outsiders but not the server itself. RFC 8827 says the defence is an independent way to verify keys, such as its identity mechanism.

SFrame (RFC 9605, August 2024) closes the SFU gap. Each sender encrypts whole frames or packet payloads with keys the SFU never gets. The SFU still sees the RTP headers it needs for forwarding, but not the faces or voices.

SFrame does not define how keys are shared; the application must provide that. RFC 9605 describes MLS (Messaging Layer Security), a group key exchange protocol, as one way to do it.

• Join control: each consult has a one-time link, and the doctor admits people from a waiting room.
• Short-lived tokens: the signaling service gives each person a token valid only for that room and a few hours.
• Recording consent: the recorder joins only after the patient taps yes, and everyone sees a recording badge.
• Logs without content: call logs keep times, quality and room ids, never video or chat text.

End-to-end encryption has a cost. A server that cannot see the video cannot record it, add captions, or mix it for dial-in. The team turns it on for consults that will not be recorded, and says so in the app.

Explain it as two paths, signaling and media, then justify an SFU with stream maths and finish with scaling and security. Here is the order that keeps the answer clear.

1. 1

   Ask about the room size

   Two people, ten or a thousand watchers change everything. Also ask about devices, networks and whether calls are recorded.

2. 2

   Split signaling from media

   Draw a signaling server over WebSocket for offers, answers and ICE candidates. Draw media as a separate path.

3. 3

   Explain NAT and TURN

   Say that STUN finds the public address and TURN relays when nothing else works. Offer TURN over TLS on port 443 as an option for strict firewalls.

4. 4

   Do the stream maths

   Count uploads per person for a five-person mesh. Then show how an SFU cuts that to one upload carrying simulcast encodings.

5. 5

   Compare SFU and MCU

   Pick the SFU for normal calls. Use mixing only for dial-in audio, recordings and very weak devices.

6. 6

   Scale and secure it

   Keep a room on one SFU, cascade across regions and drain before scaling down. Finish with DTLS-SRTP per hop and SFrame for end to end.