OpenSSL 4.1 Explained: DTLS 1.3 for UDP and What Breaks on Upgrade

TLS (Transport Layer Security) breaks over UDP because it expects every byte to arrive once and in order, and UDP promises neither. OpenSSL, the open-source library that many servers and apps call to encrypt their connections, ships a UDP version of TLS called DTLS. OpenSSL DTLS support reaches version 1.3 in OpenSSL 4.1.0.

TLS is the protocol behind the padlock in your browser. It normally rides on TCP, which resends lost data and puts it back in order before TLS sees it.

UDP sends separate packets, called datagrams, and forgets them. Apps that hate waiting use it: RFC 9147 names media streaming, internet telephony and online gaming. A voice packet that arrives late is useless, so these apps would rather skip it than wait.

RFC 9147, the DTLS 1.3 standard, lists four reasons TLS cannot run straight on UDP. The first: each TLS record (one chunk of protected data) is decrypted using a counter both sides keep in step without ever sending it. Lose one record and every later record is decrypted with the wrong count.

The second is the handshake, the opening exchange that sets up keys. TLS runs it in lock-step, so one lost message leaves both sides waiting. The other two: handshake messages can be bigger than one datagram, and faked source addresses invite denial-of-service abuse.

DTLS (Datagram Transport Layer Security) is TLS reworked to survive all four. Each fix shows up later in this guide: record numbers, resend timers, fragments and cookies.

Picture a common setup. A team runs the media servers for a video-call app, and in internet calls DTLS agrees the keys while a companion protocol called SRTP encrypts the audio and video. The setup is made up, but every command and output in this guide comes from real runs of OpenSSL 4.1.0.

Their servers speak DTLS 1.2 today. Before moving to OpenSSL 4.1, the team needs three answers: what DTLS 1.3 changes on the wire, what happens when packets get lost, and what breaks in the upgrade.

You need seven words. Each one comes up again in the traces and tests below.

• Datagram: one UDP packet, sent on its own. It can arrive late, twice, out of order or never.
• Handshake: the opening exchange where both sides agree a version, check the server's certificate and derive fresh keys.
• Flight: a group of handshake messages that one side sends together before it waits for the other side.
• Record: one chunk of protected data with a small header in front. One datagram can carry several records.
• Epoch: a number that says which set of keys protects a record. It goes up each time the keys change.
• Cookie: a token the server computes from the client's address and sends back. The client must echo it, which proves it really receives packets at that address, and the server need not remember it.
• MTU (maximum transmission unit): the largest packet a network path carries in one piece. Bigger handshake messages must be cut into fragments that fit.

OpenSSL DTLS 1.3 is OpenSSL's implementation of RFC 9147, the DTLS version built on TLS 1.3. It gives UDP apps the TLS 1.3 handshake and ciphers, plus the extra parts UDP needs: cookies, record numbers, ACKs and resend timers.

RFC 9147 was published in April 2022 and obsoletes RFC 6347, the DTLS 1.2 document, though OpenSSL 4.1 still speaks DTLS 1.2. Before 4.1.0, OpenSSL stopped at DTLS 1.2, so this is the first stable OpenSSL with DTLS 1.3.

The 4.1.0 release notes list five new features.

• DTLS 1.3, with a new manual page called ossl-guide-dtlsv13.
• GREASE from RFC 8701, an opt-in client feature explained below.
• DTLS support in the listener API, so one UDP socket can accept many clients.
• IKEV2 KDF, a key derivation function from the IKEv2 key exchange protocol.
• Initial support for Elbrus2000 (e2k) processors.

The next section traces a real DTLS 1.3 handshake, message by message. Watch for three changes from DTLS 1.2: a hidden certificate, ACK messages and a much bigger first message.

The release also removes two build options and changes how certificates are matched to hostnames. Those changes are covered in the upgrade section, because they can break working systems.

A DTLS 1.3 handshake is a TLS 1.3 handshake with a cookie check in front and ACK messages at the end. We traced one between two OpenSSL 4.1.0 programs on one machine, and the diagram shows each message that crossed the wire.

First the client sends a ClientHello, its opening offer of versions, ciphers and keys. The OpenSSL test server answered with a HelloRetryRequest (a request to try again) carrying a cookie. The client then sent its ClientHello again with the cookie attached.

The cookie check matters because UDP source addresses are easy to fake. Without it, an attacker could make a server send its large reply to a victim's address. RFC 9147 also asks servers to send at most three times what an unverified client sent.

The server's reply mixes two epochs. ServerHello travels in epoch 0, in plain text, because no keys exist yet. Everything after it, including the server's certificate, travels in epoch 2 under handshake keys.

Epoch 1 is kept for early data, which a returning client may send before the handshake ends. Our run sent none, so epoch 1 never appeared.

The ACK is new in DTLS 1.3. Each side ends its handshake with a Finished message, which proves both saw the same exchange. Over TCP the client's Finished needs no reply, but over UDP the client cannot tell whether it arrived.

So the server sends an ACK. The client ACKs each NewSessionTicket the same way; a NewSessionTicket is a ticket that lets the client resume faster next time.

RFC 9147 lets a DTLS 1.3 client send application data straight after its own Finished, as TLS 1.3 does. DTLS 1.2 waits longer: in our 1.2 run, the client's first data left only after the server's Finished arrived.

DTLS 1.3 survives loss with two separate tools. Every record carries its own epoch and sequence number, and lost handshake flights are resent on a timer. Lost application data is simply gone: RFC 9147 says DTLS does not compensate for lost or reordered data traffic.

Think of a long letter posted as separate pages. Each page says its chapter (the epoch) and its page number (the sequence number), so the reader can open any page alone. If the first pages never arrive, the sender posts them again after a wait.

Where the picture breaks: DTLS also hides the page numbers from anyone reading the envelopes. It reposts only handshake and post-handshake messages, such as a NewSessionTicket, never the pages of your actual data.

What the unified header carries

Encrypted DTLS 1.3 records start with a short unified header. In OpenSSL it does the job of DTLS 1.2's 13-byte header in 5 bytes, and it packs the flags and part of the epoch into its first byte.

The sequence number in that header is encrypted with a mask, so an onlooker cannot count your records. The receiver removes the mask, rebuilds the full number and decrypts each record on its own. A sliding replay window then drops duplicates, so an attacker who records a packet and sends it again (a replay) gets nowhere.

What happened when we dropped two packets

We put a small UDP relay between client and server that could drop chosen datagrams. In one run it dropped two pieces of the server's ServerHello. The test server had cut it into small fragments. On our loopback test it fell back to OpenSSL's 256-byte minimum link MTU. Take away 28 bytes of IPv4 and UDP headers, and no datagram carried more than 228 bytes.

The client did not ACK the pieces it did get. It waited, then resent its whole ClientHello 1.002 seconds after the start. The server sent its full flight again, and the handshake finished about 1.01 seconds in.

Nobody asked for the two lost pieces by name. The client resent its own last flight, and that made the server resend its whole flight. Losing two pieces cost one full second and a resend of everything. A lost media packet is simply skipped, but a lost handshake piece stalls the call before it starts.

That one-second wait is OpenSSL's starting resend timer. It doubles after each timeout, up to 60 seconds, which matches RFC 9147's advice. In code, DTLS_set_timer_cb lets you replace it.

The 4.1.0 listener demo warns in a comment that this default backoff can keep a handshake with a dead peer retrying for nearly 8 minutes before it gives up. The demo caps each wait at 8 seconds. For a call app, a cap like that is worth copying.

GREASE (Generate Random Extensions And Sustain Extensibility, RFC 8701) makes a client add reserved junk values to its ClientHello. Servers must ignore values they do not know, and GREASE keeps checking that they do.

The reason is ossification. If servers only ever see today's values, some get written to fail on anything new, and a future TLS feature then breaks connections. Random reserved values test that tolerance on every connection.

In OpenSSL 4.1.0, GREASE is off by default. Turn it on with SSL_OP_GREASE in code or -grease on s_client. We counted zero GREASE values without the flag and seven with it, over both TLS 1.3 and DTLS 1.3.

The seven spots were cipher suites, supported groups, signature algorithms, supported versions, the key share and two dummy extensions. The values changed per connection: one run used {0x0A, 0x0A}, another {0x1A, 0x1A}. Both handshakes completed.

RFC 8701 binds both sides. Servers must never pick a GREASE value, and clients must reject one if a server picks it anyway.

On the wire, DTLS 1.3 hides more, confirms more and sends bigger keys. The table compares two runs on the same OpenSSL 4.1.0 build, one pinned to each version.

The bigger DTLS 1.3 handshake is mostly the hybrid key share, and it travels twice. The client's 1,216-byte share sits in both ClientHellos, before and after the cookie, which were 1,460 and 1,586 bytes.

On UDP that means more fragments and more pieces to lose, so the path MTU matters more than before.

For the media-server team, the plain-text certificate row is the privacy win. Anyone watching DTLS 1.2 traffic can read which server certificate was used; with DTLS 1.3 they cannot.

Their keying step still works too. With -use_srtp SRTP_AES128_CM_SHA1_80 on both sides, s_client reported SRTP Extension negotiated over DTLS 1.3 on 4.1.0.

What the unified header carries, bit by bit

The first byte of an encrypted DTLS 1.3 record packs three fixed bits 001, a Connection ID bit, a sequence-length bit, a length bit and the epoch's low two bits.

OpenSSL always sends a 2-byte sequence number and a 2-byte length, so its header is 5 bytes. In our trace, handshake records began with byte 0x2E (epoch bits 2) and epoch-3 records, such as ACKs and application data, with 0x2F (epoch bits 3).

Build 4.1.0 into a private folder, start s_server in DTLS mode and connect with s_client. These are the exact commands we ran on Linux on 26 October 2026, with the output to expect.

1. 1

   Check the download

   Save openssl-4.1.0.tar.gz and openssl-4.1.0.tar.gz.sha256 from the GitHub release page in your home folder. Expect openssl-4.1.0.tar.gz: OK from this command: sha256sum -c openssl-4.1.0.tar.gz.sha256

2. 2

   Unpack the source

   Run tar xzf openssl-4.1.0.tar.gz and then run cd openssl-4.1.0

3. 3

   Configure a private install

   This keeps 4.1 away from your system OpenSSL. Run ./Configure --prefix=$HOME/ossl41 --libdir=lib

4. 4

   Build and install

   The install_ssldirs target adds openssl.cnf, which install_sw alone skips. Run make -j4 and then run make install_sw install_ssldirs

5. 5

   Point your shell at it

   Run these three commands one by one. Expect OpenSSL 4.1.0 23 Oct 2026 (Library: OpenSSL 4.1.0 23 Oct 2026); a 3.x answer means later steps will fail. export PATH=$HOME/ossl41/bin:$PATH then export LD_LIBRARY_PATH=$HOME/ossl41/lib then openssl version

6. 6

   Make a test certificate

   Run mkdir -p $HOME/dtlstest and then cd $HOME/dtlstest to get a clean folder. Then run this as one line: openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes -keyout key.pem -out cert.pem -days 30 -subj /CN=localhost -addext subjectAltName=DNS:localhost

7. 7

   Start a DTLS server

   The -dtls flag means UDP with any DTLS version, and the highest shared version wins. Run openssl s_server -dtls -accept 127.0.0.1:14432 -cert cert.pem -key key.pem

8. 8

   Connect with DTLS 1.3

   Open a second terminal and run the two export commands from the earlier step again. Expect New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384 and then Protocol: DTLSv1.3, and press Ctrl+C to leave. Run openssl s_client -dtls1_3 -connect 127.0.0.1:14432

9. 9

   Read the right line

   We saw Protocol: DTLSv1.3 printed after a failed handshake too. Cipher is (NONE) means nothing connected. Passing -dtls and -dtls1_3 together fails with Cannot supply multiple protocol flags.

10. 10

    Compare with DTLS 1.2

    Leave the server running. Expect New, TLSv1.2, Cipher is ECDHE-ECDSA-AES256-GCM-SHA384 and Protocol: DTLSv1.2 from this command: openssl s_client -dtls1_2 -connect 127.0.0.1:14432

11. 11

    Look inside with -trace

    Look for DTLS 1.3 (65276) under supported_versions, cookie_ext in the first server reply, and Inner Content Type = UNKNOWN (26), which is the ACK. Run openssl s_client -dtls1_3 -trace -connect 127.0.0.1:14432

12. 12

    Try GREASE

    Expect UNKNOWN entries such as {0x0A, 0x0A} in the ClientHello trace, and the same Cipher is line as before. Run openssl s_client -dtls1_3 -trace -grease -connect 127.0.0.1:14432

The trace prints ACKs as UNKNOWN (26) because its printer has no name for the new content type. That label is undocumented as of 26 October 2026, so pin your version if a script greps for it.

The biggest surprise is not DTLS: OpenSSL 4.1 stops matching hostnames against a certificate's Common Name. If a certificate has no DNS subject alternative name (SAN, the field listing valid hostnames), hostname checks now fail.

This bites only where your code asks OpenSSL to check a name, for example with SSL_set1_dnsname or -verify_hostname. The older SSL_set1_host has been deprecated since OpenSSL 4.0. The same change stops email checks from falling back to the emailAddress field of the subject.

We tested a certificate with CN=device.example.test and no SAN. OpenSSL 4.1.0 printed error 62 at 0 depth lookup: hostname mismatch. The system's OpenSSL 3.0.13 accepted the same file, and a copy with a SAN passed on both.

Code can bring back the old behaviour by passing X509_CHECK_FLAG_ALWAYS_CHECK_SUBJECT, from openssl/x509v3.h, to SSL_set_hostflags or X509_VERIFY_PARAM_set_hostflags. Both replace the existing host flags, so OR it into any you already set. In our test that made the CN-only certificate pass again. We found no switch for it in openssl verify -help.

The lasting fix is to reissue certificates with a SAN. Call media may escape this: in DTLS-SRTP as RFC 5763 defines it, peers check the certificate against a fingerprint sent in the call signalling, not a hostname.

OpenSSL's own new DTLS listener demo trips on this. Its bundled cert.pem has CN=localhost but no subject alternative name, so its DTLS 1.2 and 1.3 clients both failed with certificate verify failed. Its README still says to connect as localhost. This is not in the release's known-issues list as of 26 October 2026.

Check your support window too. As of 26 October 2026, OpenSSL 3.4 support has ended (22 October 2026) and 3.6 is due to end on 1 November 2026.

OpenSSL 4.0 is supported to 14 May 2027 and 3.5 LTS to 8 April 2030. No end date for 4.1 is published yet.

Mixed fleets still interoperate. In our test, an OpenSSL 3.0.13 client still got DTLS 1.2 from a 4.1.0 server, and a 4.1.0 client fell back to DTLS 1.2 against a 1.2-only server.

OpenSSL 4.1 extends its listener API to DTLS. SSL_new_listener wraps one UDP socket, checks each new client's address with a cookie, and hands you one connection object per client through SSL_accept_connection.

This matters because the older DTLSv1_listen call supports only DTLS 1.0 and 1.2 with HelloVerifyRequest. It cannot do DTLS 1.3, so a DTLS 1.3 server that shares one port needs the listener.

Address checks are on by default. SSL_LISTENER_FLAG_NO_VALIDATE turns them off, which the manual does not recommend on untrusted networks. The defaults are 256 pending connections, a 30,000 ms pending timeout and 2,000-byte maximum datagrams.

We built the dtlslistenerecho demo from the 4.1.0 source and gave it a certificate with a SAN. A DTLS 1.3 client and a DTLS 1.2 client stayed connected together to UDP port 4433 for six seconds, and both got their messages echoed.

Our only code change was IPv4 instead of IPv6, because our test machine had no IPv6. Two of the known issues in the next section, 33277 and 33278, sit in this listener path. Until fixes ship, log its error codes before acting on them.

Four DTLS problems are listed as known issues in the 4.1.0 release notes, and the manual names a few limits. None stopped our basic tests, but each one matters in production.

• Issue 32878: when a DTLS 1.3 KeyUpdate (a mid-connection key change) arrives, other unconfirmed post-handshake records, such as a NewSessionTicket, leave the resend buffer. A fix is planned.
• Issue 33242: a DTLS 1.3 server may abort with an unexpected_message alert when the client ACKs part of the server's first flight. Packet loss can cause this.
• Issue 33277: a permanent receive failure on a DTLS listener can be reported as SSL_ERROR_WANT_READ, so the app keeps waiting.
• Issue 33278: a retryable write failure on a nonblocking listener connection can be reported as SSL_ERROR_SYSCALL, so the app may close a usable connection.

Connection IDs are also missing. A Connection ID lets a server find a session after the client's IP address or port changes, for example when a home router remaps it (NAT rebinding).

OpenSSL 4.1.0 does not support them, and its record code silently drops records that carry one. RFC 9147 applies the Connection IDs of RFC 9146 to DTLS 1.3.

RFC 9146 explains the cost of lacking them: an address change means a new handshake. For a call app, a phone that moves from Wi-Fi to mobile data must handshake again.

The manual lists two more limits. DTLS over SCTP still negotiates DTLS 1.2 or older, and the short-tag CCM_8 suite is refused for DTLS, as the upgrade table shows.

A wrong version number in the manual and a hidden 256-byte MTU

The ossl-guide-dtlsv13 manual says DTLS 1.3 adds the value 0xFEFD to supported_versions. That is DTLS 1.2's number. The header file, RFC 9147 and our trace all show 0xFEFC (65276), so treat the manual line as a documentation error as of 26 October 2026.

Back to the small fragments: in our loopback test, s_server sent datagrams of at most 228 bytes, because it fell back to OpenSSL's 256-byte minimum link MTU. With -mtu 1400 its flights shrank from 15 datagrams to 6. This fallback is undocumented as of 26 October 2026; pin your version and set the MTU explicitly.

Frame it as TLS 1.3 adapted to a transport that loses packets, then show what each piece protects against. The call media server makes a good running example.

1. 1

   Start with the transport

   Calls and games use UDP because late data is worthless. TLS needs reliable, ordered bytes, so it cannot sit on UDP unchanged.

2. 2

   Walk through the handshake

   A cookie round trip proves the client's address. Then comes a TLS 1.3 handshake split across epochs 0 and 2, closed with an ACK.

3. 3

   Explain loss handling

   Records carry epoch and sequence numbers, so each one decrypts alone. Lost handshake flights are resent on a timer, which starts at 1 second in OpenSSL. Lost media is never resent.

4. 4

   Size the packets

   Hybrid post-quantum key shares push handshake messages past 1,000 bytes. Set the path MTU so fragments fit, and expect more fragments on small-MTU links.

5. 5

   Scale the server

   Use one UDP port with a listener that separates clients by address. Without Connection IDs, a client whose address changes must handshake again.

6. 6

   Close with operations

   Cap the resend backoff, pin library versions, watch the four 4.1.0 known issues and confirm every certificate has a SAN before upgrading.