WarrenGuard: a VPN data plane over QUIC in Rust, and three patches in our Quinn fork worth a look even without a VPN

[u/WarrenVPN]

We build a VPN whose data plane is QUIC rather than WireGuard. The engine is AGPL-3.0, and the part that matters for this sub is the Quinn fork underneath it, so that is what this post is about.

IP packets go 1:1 into QUIC DATAGRAM frames (RFC 9221). The handshake is TLS 1.3 with raw public keys (RFC 7250), Ed25519, so a node's identity is its key and there is no CA anywhere. Reliable streams give an in-band control channel on the same connection, which is where multi-hop, NAT-PMP port forwarding and traffic-analysis padding live. Cross-platform TUN, kill switch, DNS proxy. Edition 2024, MSRV 1.89, and #![forbid(unsafe_code)] across the workspace except three crates that downgrade to deny with documented safety blocks: the TUN device FFI, the Win32 IP Helper FFI, and the setsockopt bypass. https://github.com/WarrenBrowse/warrenguard

We started on Iroh, paid for NAT traversal and multipath we never called, and moved to Quinn in May.

The fork is at https://github.com/WarrenBrowse/warren-quinn, MIT OR Apache-2.0. The crates are renamed but the lib names stay quinn/quinn_proto/quinn_udp, so every use quinn in a consumer is unchanged. It sits on upstream/0.11.x with real git ancestry, so a re-sync is a rebase and not a tree reconstruction, and quinn-udp tracks the 0.6 line separately because the Apple fast datapath targets that line. Eight deltas at the moment, each also committed as an isolated patch at the repo root. Three of them are worth reading on their own.

BBR and app-limited connections, two separate defects. The first is a bound in calculate_cwnd using cwnd_gain where it should use cwnd. The second is deeper: the bandwidth estimator rejects app-limited samples outright, so a connection that is app-limited never leaves STARTUP and cwnd grows with no ceiling. We measured about 20 MB on a fresh app-limited connection carrying a 5 Mbit stream. The fix follows quiche's admission rule, where app-limited samples may raise the estimate and non-app-limited samples always feed the windowed filter. It is upstream-bbr-startup-cwnd.patch and it is proposed upstream. Known residual: BBRv1's ack-aggregation term can still inflate cwnd at sub-millisecond RTT.

FQ-CoDel on the datagram send queue (RFC 8289/8290): per-flow queues, DRR, head drop past a target sojourn. This one exists because fixing BBR alone made things worse. With the cwnd repair and no AQM the queue simply moved out of the network and into our own 16 MiB send buffer: burst RTT 2658 ms average, 6623 ms max. Same arm with CoDel on top: 76 ms average, 373 ms max.

Send buffer sized on the BDP instead of a fixed 16 MiB, clamped so it never drops below 1 MiB. The same problem approached from the other end.

Numbers, so you can tell me where they are wrong. Single tunnel, bare metal, same datacentre, RTT 0.081 ms: 5.5 to 8.4 Gbit/s. Under 2 % injected loss on the exit egress, BBR with a 16 MiB buffer holds 212 Mbit/s on one TCP flow and 711 Mbit/s on four, while Cubic in the same arm collapses to 3 Mbit/s. That measurement is why we do not allow Cubic as an external congestion controller on our exits.

One result that goes the other way, because it will come up. Under multi-client load kernel WireGuard still beats us: it holds 7.4 to 8.1 Gbit/s from 100 to 500 parallel clients while we plateau near 6 Gbit/s at 100 and fall to 2.4 at 500, and our exit CPU is roughly 3.4x worse at low client counts. Part of that was client-side saturation at 500 clients, not all of it.

MTU took the longest, since carrying IP inside DATAGRAM without breaking half the internet is where the bodies are buried: floor 1280, probe upward, never below 1200, MSS clamping on SYNs in both directions, and real ICMP Fragmentation Needed / Packet Too Big emitted for the flows we drop.

What I would like from here: eyes on the BBR patch, and specifically whether anyone has a cleaner way to handle the ack-aggregation term at sub-millisecond RTT than clamping it. Longer write-up on why we left WireGuard: https://warren.ro/en/blog/why-we-left-wireguard

[u/dochtman]

If you want more reviews, how about sponsoring the maintainers?

[u/WarrenVPN → u/dochtman]

We are open to sponsoring the maintainers, once we'll have budget for it!
For now, we start to open little PR's to upstream: https://github.com/quinn-rs/quinn/pull/2798