Why We Love QUIC and HTTP/3
fastly.com
fastly.com
Let's compare a 100MB file served via TCP to the same file served via QUIC.
TCP:
- web server sends 2MB at a time, 50x times, via async sendfile (50 syscalls & kqueue notifications)
- kernel reads data from disk, and encrypts. The data is read once and written once by KTLS in the kernel.
- TCP sends data to the NIC in large-sh chunks 1.5k to 64k at a time, lets say an average of 16k. So the network stack runs 6250 times to transmit.
- The client acks every other frame, so that's 33333 acks. Let's say they are collapsed 2:1 by LRO, so the TCP stack runs 16,666 times to process acks
QUIC:
- web server mmaps or read()'s the file and encrypts it in userspace and sends it 1500b at a time (1 extra memory copy & 66,666 system calls)
- UDP stack runs 66,666 to send data
- UDP stack runs 33,333 number of times to receive QUIC acks (no idea what the aggregation is, lets say 2:1)
- kernel wakes up web server to process QUIC acks 33,333 times.
- 4x as many network stack traversals due to the lack of TSO/LRO.
- 1000x as many system calls, due to doing all the packet handing in userspace
- at least one more data copy (kernel -> user) due to data handling in userspace.
Disclaimer: I work on CDN performance. We've served 90Gb/s with a 12-core Xeon-D. To serve the same amount of traffic with QUIC, you'd probably need multiple Xeon Gold CPUS. I guess that Google can afford this.
100k is not that much different than 100mb, except the TCP window will not be as far open, so TSO will not be as effective.
Note that I work on a CDN that serves large media files, so I'm biased towards that workload.
But again, couldn't there be NICs with offloading QUIC capabilities? Maybe this could even be done with firmware updates (I don't know how much of the TCP offloading is done in real hardware)
sendmmsg (or the upcoming io_uring) let you send multiple UDP packets with a single syscall.
Can you explain more about how the negatives you mention weigh up against the positives? There isn't a net benefit somewhere? If not, can something be changed to give a better balance like a hybrid solution?
A good document outlining considerations can be found here: https://http3-explained.haxx.se/en/
Even HTTP/2 involves some of the issues the parent mentions. HTTP/2 is not really helpful for large files, and might likely perform worse than HTTP/1.1 due to the additional insertion and parsing of control flow headers. HTTP/2 helps small files most, by avoiding the overhead of connection establishment for those.
These aren't intrinsic problems with QUIC, they're common to all new protocols.
I also don't think a QUIC server would read the whole file into user-space at once - that's just a giant memory waste. Rather it would be streamed and chunks would get encrypted. That process requires of course an extra copy (likely even two for the unencrypted and encrypted version), but that's the same for all user-space file serving and encryption options and nothing new due to QUIC. For KTLS it would need to get investigated whether the kernel solution doesn't also perform a copy somewhere (I honestly don't know).
Having written the FreeBSD kernel TLS, I can assure you that there is no copy. Data is brought into the kernel via DMA from storage into a page in the VM page cache. When the IO is done, it is then encrypted into an connection-private page. That page is then sent and DMA'ed on the network adapter. So we have in the kernel tls case:
- memory DMA to kernel mem from storage.
- memory READ from kernel mem to read plaintext for crypto
- memory write to another chunk of kernel mem to write encrypted data
- memory DMA from kernel mem to NIC
For QUIC you have:
- memory DMA to kernel mem from storage
- memory read from kernel mem via mmap
- memory write to userspace mem to write encrypted data
- memory read from userspace mem to copy to kernel
- memory write to kernel mem
- memory DMA from kernel mem to NIC
Right now, we can just barely serve 100g from a Xeon-D because Intel limited the memory bandwidth to DDR4-2400. At an effective bandwidth limit of 60GB/sec, that's on the edge of being able to handle the kernel TLS data path. So even if everything else about QUIC was free, this extra memory copy from userspace would cut bandwidth by a third.
Additionally, since QUIC streams allow for client IP mobility, that creates an additional challenge for IP level load balancing as well as handling at the host level. In a well configured host, TCP packets for a given stream will always arrive at the same nic queue, on the same CPU, allowing the TCP data structure to be local to that CPU and avoid cross-cpu locks. In QUIC, the next packet can come from a new IP, which could be ECMP routed to a different host, or arrive on a different NIC queue and a different CPU. Perhaps, your ECMP router and NIC can be taught to look for the QUIC connection IDs, but that doesn't seem at all certain.
QUIC/HTTP3 as a protocol is a great improvement to actual internet performance for users which is what really matters.
Things don't automatically get better. It is hard work, it sucks, and it's not for everyone. It will take years to undo the damage of this transition. We will still be working on it in a decade. There are some very subtle gains like HOL-blocking. I'm not convinced that outweighs current actualized improvements in TCP congestion control (BBR), and for any application I can think of the places that really need something message-oriented seem better covered by WebRTC.
What you are really talking about is Full Employment Theorem.
What "damage" are you talking about? The only issues are compatibility and increased resource utilization on the server-side, both of which will get better as usage increases. It's not a problem. We go through these cycles all the time with all kinds of technology and there's nothing special here.
QUIC is a new protocol is to make user experiences better. There's a tradeoff in more server CPU but that's cheaper, more scalable, and will only be short-term as things quickly improve. The actual comparison would be rendering engines and Javascript runtimes that have become more complicated to build and run but are faster and more functional in return.
Nobody would return back to the 2010 tech days just because some people decided to make fat websites.
There are also more general efforts underway on Linux to reduce the system call and copying overhead of processing packets in userspace. TPACKET_V3 is an easy way to vastly increase the scalability of UDP recv processing with minimal application changes. AF_XDP is much more extreme, but it is going to be more implementable than the older DPDK-style semantics. It effectively will put packet buffer management into userspace with the transport. But once you're doing that have recaptured much of the advantages that TCP has by running in the kernel.
Anyways, since QUIC is being touted everywhere as being very secure:
> [QUIC] protects both the data and the transport protocol itself
It seems like missing ESNI as a required feature is a bit of a glaring omission. Does anyone have a better understanding? To me, it seems like a great opportunity to make ESNI required for HTTP/3. Much like how browsers made TLS required for HTTP/2. I would love further insights if anyone has any.
If I understand it correctly, I think that countries might start to block ESNI altogether. If it is not widely implemented, websites/apps using it will standout which sadly could limit its adoption. For instance, if Signal decided to use ESNI, it will probably get blocked in those countries, but this can change if big companies wanted to use it. However, I still don't know how it will work exactly.
QUIC uses the TLS 1.3 client hello (and its extension mechanism) for the handshake, so evolutions to TLS 1.3 like ESNI will be automatically valid in QUIC too as they come down the pike.
In any case, it's hard for a client to determine if a TXT record is not present, or unavailable because the authoritative server has no such record, or because something in the middle has blocked it (due to network incompetence or active malice), so if you want that to work, you're going to need to specify dns over https to a trusted third party, and de-decentralize DNS.
That said, from a brief look at the spec, the ESNI extension includes a digest of the key record, so while an observer can't directly read the SNI, given a sufficient effort to find the keys, they could correlate the digests with matching hostnames.
Reaching key agreement to exchange identity information without disclosing the identities in the clear is somewhere between really hard and impossible.
I think you've probably misunderstood what's going on here
ESNIKeys, the data structure you're talking about, isn't a key for a specific name, it's the key for the frontend server that's agreed to do ESNI and can be used for ALL names offered on that frontend server.
Whatever name you're asking about, you get the same ESNIKeys values, whether you wanted cat-photos.example.org or nazi-death-squad.example.net or boring-corporate.example.com if they are all hosted on 10.20.30.40 (or that's a TLS load balancer for perhaps different backends that don't face the outside world) they all have the same ESNIKeys.
The fact they're shared is why there's a length field. We can only safely protect names by padding them. Otherwise it doesn't take a genius to spot that this-very-long-name.subdomains-matter-too.example.com encrypt to a far larger structure than short.example. The length field says I promise all the names I'm protecting with ESNI will fit in a name structure this long, just pad the shorter ones.
The digest is in the ESNI setup because this way a server can go "Oh, you've got last week's keys somehow. No, those won't work" or equally "Those are my Cloudflare keys! We only use Cloudflare in North America, this is a European server, we do AWS here, why have you got those?". Without a digest you have no clue why this idiot client is sending you gibberish and you can't do diagnostics.
[1] https://datatracker.ietf.org/meeting/104/agenda.html
[2] https://datatracker.ietf.org/doc/draft-ietf-quic-transport/
Much like in TCP, congestion control really isnt something required for interoperation between peers. Given the userspace nature of QUIC I would expect to see a lot of iteration on this front - for good and bad. (but hopefully the bad iterates quickly).
The current drafts describe newreno is detail, but also explicitly call out the ability to run other things. I've seen reno, cubic, and bbr all run with quic and anticipate others to happen as well. That's one of the exciting things here.
QUIC is the tool to bring about the next phase of Internet centralization by the mega players.
It is that it uses UDP, which means UDP cannot be blocked by simple ACLs. Blocking all UDP is a simple technique for avoiding other protocols that allow for reflection DDoS attacks.
I haven't got experience with how effective blocking UDP is as a mechanism for avoiding DDoS, but it does sound pretty simple and easy to deploy.
IPv6 is better since there is no fragmentation. Maybe QUIC/http3 should be IPv6 only?
There is nothing worse than finding out that someone not even at the company anymore decided years before to deploy some crap like this. Drives me absolutely crazy to impose stuff like this where silos in companies means transitioning involves on the order of 4-5 different "components" need to change.
There isn't much opportunity for people to plug random stuff between your server and the middlebox (the main middlebox would disallow it, like anything else), but there is still plenty of crappy rules everywhere and nobody knows why they exist or what they are. And you can't even call your ex-coworker and ask for help, because it's an ex-employee of the middlebox company, not yours.
Anyone remember TTCP?
If they don't have a valid cookie Fast Open costs the same as regular TCP in the face of adversaries trying to DOS you. You examine the packet, it doesn't have a valid cookie, you discard it. No further work, just like ordinary TCP.
I hope that QUIC has used these lessons from TCP and TLS to make changes in the future as easy and effective as possible, but I’m sure it’ll still have its limits.
Hardly sounds useful.
TCP Fast Open was standardized 8 years ago and is barely used. This is because updating TCP requires kernel updates, which just isn’t going to happen on most mobile devices.
Thus, moving the protocol to userspace makes a lot of sense.
1. Our design is complete, error-free and designed to stand the tests of time. It will be in common use, largely unchanged, in 40 years time. The TCP of the 2010s. There is widespread industry support. We want to move to the application layer to ease the initial roll-out.
2. Our design will need to change every few years, even we authors don't think it's finished. This is a Google-only project and most vendors are refusing to support it as they think it's badly designed crap. The ActiveX of the 2010s. We want to move to the application layer so we can force it through without anyone else's support.
Where are we on the spectrum between those two options? I don't know.
So no, it’s not the same.
Another issue they mention in the article is middleboxes, which basically will never be upgraded, and will never support new TCP features.
Raw IP sockets are accessible from the same userspace facing APIs as e.g. UDP sockets and don't require climbing up the stack. Unfortunately operating systems started to consider custom protocol implementations security risks but rather than reverse that thinking we've just continued to abstract up past it.
In reality I think "where it is implemented in code" was a small portion of QUICs design choices compared to "IPv4 NAT & external firewalling has ossified protocols" which is a similar story of "just abstact up to avoid the issues". Unfortunately in that case I don't think abstracting up isn't as permanent a solution as it was on the OS side.
Likewise NAT devices typically support UDP flows today due to their prevalence in games, but if you introduce a new transport protocol at the IP layer, they wouldn't be able to identify which flow (and therefore which NATed endpoint) the packet is destined for.
Though I don't know what's wrong with UDP. 8 bytes of overhead for 1450 bytes IP payload is 0.5% bandwidth. Checksum overhead should be negligible.
That's expensive. Even if you avoid copying the packages by sharing memory between processes, there are still a lot of context switches...
In reality raw sockets work in a way that the question is the reverse of what you describe. The kernel will check 2 things: - Which raw sockets are bound to the protocol number seen in the packet - Which raw sockets have issued "connect" to the sending IP
Any and all raw sockets that match these will receive the packets. In such a sense the protocol (QUIC) needs to have some way to identify streams so that if e.g. both Chrome and Firefox browse to the same server they don't interfere with each other. QUIC innately has this functionality due to the way it implements encryption. Ideally the OS would allow a raw socket to register something akin to a BPF filter though as that would make it equally as efficient as UDP socket tracking even in the edge cases.
> Likewise NAT devices typically support UDP flows today due to their prevalence in games, but if you introduce a new transport protocol at the IP layer, they wouldn't be able to identify which flow (and therefore which NATed endpoint) the packet is destined for.
This is actually what I was referring to when I said:
> "IPv4 NAT & external firewalling has ossified protocols" which is a similar story of "just abstact up to avoid the issues"
We continue to make non choices to build up the stack rather than implement systems that are interchangeable.
The important design distinction about the layers is what element has access to what data. (e.g. routers need to see IP addresses to do their job, port numbers help kernels segment permission models, etc..) The rest of it is just about logical models, real-world workarounds, and luck...
HTTP/3 will be able to get high bandwidth, buttery responsive restarts to connections far too long idle to keep "open" because it integrates security, application, and transport. That's thoughtful design, not a workaround.
I'm going to paraphrase (and maybe bungle, because I don't have it at hand) from my favorite networking book of all time - the underappreciated _Network Algorithmics_ by George Varghese. He describes layers as a lovely way to model and think about a protocol or design, but often a terrible way to build one. I've spent a lot of time thinking about that, and I think QUIC gets it right - the layers are clear in how they inter-relate but they do so without being independent.
I actually disable QUIC myself, because I've noticed it slows everything down too much on some home routers.
I'm curious how you measured and came to this conclusion since the design and most all metrics claim the opposite? And are you sure it isn't a bufferbloat issue rather than QUIC?
Also, some routers prioritize all UDP packets because they treat them as VoIP. Then all TCP suffers.
Some service providers traffic shape UDP because uTorrent uses it (UTP) on random port numbers.
Similar arguments go for the prioritization.
Edit: Ahh...read some of your other posts. You aren't a fan of HTTP/3, which puts your comment in context for me.
Things are changing and getting a lot more difficult with HTTP/3 (IETF QUIC) and TLS1.3. Many vendors are claiming TLS1.3 support today, but the interesting thing is nobody is talking about the dismal performance implications it has on packet processing. With TLS1.3 and without HTTP/3 all sessions must use PFS for transform selection. And on top of it with 0RTT if a client gets to the server before the middlebox does - then, I believe, it becomes a failure scenario at the end user experience. Security vendors like Fortinet, Forcepoint, Palo Alto Networks and Cisco are all up against the wall long term. Consider they sell these devices for millions of dollars per device in larger variations. Now we're moving back to taking a device that claims tens to hundreds of gigabits of deep packet processing to, hundreds? They won't share the performance impact with customers - because that will impact financials, which will flow down to stock price, etc, etc. I feel as though companies that bank on the middlebox (ie NGFW) know of the impending apocalypse but are choosing, collectively, to stay quiet. Cisco did have an article that indirectly admitted this but only in context of TLS1.3 and not HTTP/3 [0].
What is the general consensus of others as we see HTTP/3 gain popularity? None of the aforementioned vendors do MitM decrypt with Google properties riding Google QUIC today, as ultimately they can't. The "security" coverage then moves to software / endpoints to pick up the pieces (where plaintext traffic is still available). But in the meantime I feel like the consumer of these products is being told nothing for the sole upside of financials. I used to be a huge proponent of NGFW and the visibility they brought. However I feel as though those devices now give a very high false sense of security as they are only able to catch very low hanging fruit and are simple to bypass [1]. I'm curious what the collective here thinks about the futures of hardware network security, and with that even SaaS based (eg ZScaler).
TL;DR If you're a CISO/CSO is it now a fools errand to continue to invest money in middleboxes with the strong stronger crypto enforcement on the horizon?
[0] https://blogs.cisco.com/security/tls-1-3-and-forward-secrecy... [1] https://http-evader.semantic-gap.de/
I wouldn't mind a progression in http transports that made corporate MITM unworkable. In the past, though, some kind of crappy loophole always makes it possible.
In my country we had many televisions labelled HD Ready when HD television first became available. Were these actually ready to play HD television? Er, no. They could however tolerate existing in a world with HD while not being HD themselves and this was what they marketed as "HD Ready".
Do you have examples where they actually do TLS 1.3?
HN discussion here: https://news.ycombinator.com/item?id=16564935
The specification says: YOU must choose RANDOM numbers otherwise bad things could happen.
[ TLS 1.3 final hides a downgrade signal in those random numbers if you appear to only speak TLS 1.2. The TLS 1.2 specification says nothing about a downgrade signal, so if you recognise the signal that means you wanted TLS 1.3 but the server has been told you wanted TLS 1.2, a downgrade attack is being attempted. Abort! ]
These famous brand middleboxes were too lazy to make random numbers, they'd just take the exact numbers the real server picked and use those. Those are random right? What could go wrong?
The result was that the TLS 1.2 Downgrade signal would get copied into supposedly "fresh" TLS 1.2 connections and trip the abort mechanism.
Just an incompatibility right? Nope. For the years that this idiocy was in those products they weren't actually delivering security, the requirement that you pick RANDOM numbers is there for a good reason - if sophisticated bad guys knew this "bug" was present in the famous brand middleboxes they could definitely have exploited this to snoop connections.
https://www.fortinet.com/blog/business-and-technology/tls-is...
Keep in mind this is just TLS1.3 and they make no claims around IETF QUIC / HTTP/3.
> The good news for Fortinet customers is FortiOS 6.2 fully supports TLS 1.3 for effective and high-performance MITM inspection.
and in contrast
> The latest version of FortiOS 6.0 not only fully supports TLS 1.2 MITM, but it also does not break TLS 1.3 when it has to negotiate down to TLS 1.2.
[ The "break TLS 1.3" they're talking about is the phenomenon I described in a cousin post in this thread, several of their competitors screwed up here ]
So that suggests that in Fortinet's case products running their 6.2 release (it's unclear to me if this is merely in Beta or actually a finished product) will actually do TLS 1.3. It's sad that they feel they can boast about the earlier 6.0 product actually working correctly (compatibility with TLS 1.3 by downgrading to TLS 1.2 is literally how everything would work if you just correctly implemented the specification, yet) when it was released in 2018, many years after the TLS 1.2 specification was finalised and in wide use.
Still you know "Better late than never".
I implemented a HTTP/2 library for .NET (https://github.com/Matthias247/http2dotnet). It took quite a lot of time and dedication to get it spec conformant. I doubt that most employers (apart from some CDN) would have allowed me spending the time to get it to that level. And yet it still has lots of potential for improvement.
HTTP/3 might be even harder (I haven't read the spec yet, but the whole UDP assembly and inclusion of encryption sounds more complicated).
Compared to that building a small HTTP/1.1 library or a framework around it is much more approachable and might be also more rewarding.
Good folks at fastly, I hope you're reading... I've been waiting very patiently for part 3 of this series for a good part of 2yrs now: https://www.fastly.com/blog/building-and-scaling-fastly-netw...