A Technology Preview of Nginx Support for QUIC and HTTP/3
nginx.com
nginx.com
From my point of view, http3 is more of a sidegrade and has no net benefit over http2.
We can make TCP fast, very fast, and the use of UDP was not a prerequisite for any of its functionality.
The "multistream" functionality of QUIC is mainly to benefit companies with big, highly loaded CDNs, and kind of obviates the economic point of squeezing multiple virtual streams into a single TCP connection as used in http2.
By throwing away TCP, they are throwing away decades of optimisations, and hardware offloading that network hardware makers made to handle TCP well.
If the talk was really about extracting single digit improvements from it, I think it would've made more sense to finally put SCTP, and DCCP to good use.
Transparent handover (when changing IP addresses)
Multipath
Multiplexing: No head-of-line blocking
Faster handshake
Enforce authentication and encryption (also of some metadata)
Forward Error Correction and other improvements for wire less networks
I wouldn't be surprised if many stacks and networks are able to deploy QUIC more easily than MPTCP.
Well, UDP does not fare better on a lot of ISPs.
UDP is very commonly blocked in almost every corporate environment I have seen, it is a pain get it enabled for applications like video streaming which benefits from UDP
It's like arguing against a developing new TLS version because people configure their firewalls to block any flavor of TLS they don't know.
It is particularly worse in consulting industry, This is usually because their customer actually controls the network/infra for the teams working with them, usually that means a lot of fragmented IT managment.
This is from my experience deploying Flash / WebRTC stack video tools in Fortune 500/2000 type orgs in last 8-9 years. One of the main reasons we moved to WebRTC from Flash early on is because the heavy friction for deploying RTMP/RTMPS(E). I can count in one hand how many of them had sensible network polices.
You may had a good experience with network policies, many many large organizations still insist on IE11 and restricted networks world around, that is the sad reality .
Again, that agreement already means you're not providing internet service, even if you provided outgoing UDP you still would be violating the end to end principle. The hardware is just a consequence of wanting to be more (rather: less) than a dumb pipe.
I guess the problem is more with getting to adopt the fixes on a large scale even if the downsides of TCP are fixed.
> It just uses UDP because all other protocols are usually blocked
UDP is also blocked on several firewalls because of the fear of UDP flood attacks, QUIC fallbacks to TCP in such cases.
QUIC eliminates the TLS handshake time on top of TCP. You can't really improve on the TCP 3-way handshake - you need a minimum of 3 messages to ensure that both sides have agreed to a connection.
You also have to question whether FEC should be implemented in layer 4, rather than implemented over the lower layers only as when needed. It seems to be terribly inefficient to implement FEC end-to-end when the packetloss is due to the last mile (or several meters for WiFi).
To me, QUIC seems to be the wrong approach to the problem. If you want a connection-oriented protocol with the advantages above, it's much better just to push through a new version of TCP with the extensions. Sure, it's going to take longer than approving a new protocol over UDP, but the potential benefits are substantial. On the other hand, there is an argument that by pushing through QUIC, there is pressure to improve TCP in those ways. We'll have to see how this pans out.
Yes there is.
TCP has a variable options range of up to 320 bits starting at the 20th octet to handle exactly feature extensions (This is exactly how Multi-Path TCP is implemented). There are also 3 reserved bits set for future use (13th octet), which can also form the basis of even larger extended option ranges for stuff like TLS negotiation.
One of the lessons of the last decade or so has been that only end-to-end encryption prevents ossification. To the extent that middleboxes can read stuff they will break forward compatibility.
For example certificate compression. How are we only sending that to standardisation now in 2020? Well it was impossible to deploy this before TLS 1.3. Why? Because older versions of TLS didn't encrypt certificates, and so middleboxes could read them, and so middleboxes would freak out if the certificate wasn't as expected. That's all it took to make a useful optimisation impossible to deploy.
Adding features to TCP was done many times successfully already. When my jurney started there was no ws-scaling, ECN, SACK... They are all widely adopted now.
So what did they do? Well they adjusted ECN so that it tolerates the broken middleboxes. That's what you have today. Your systems go "Oh, ECN is broken, oh well" and press on without it. You see this as a feature added "successfully" and I call it what it is: Failure.
One way to think about it is negotiating a connection vs what that connection actually is are pretty separate. Sadly even the literature and education materials on TCP is pretty poor at explaining this. With the QUIC spec this just feels much clearer.
You can not really have "flexible ordering" with default Quic, since Quic Streams are inherently ordered Streams. You could treat them as unordered on one end and deliver out-of-order data as soon as at arrives, but that won't allow the peer to send unordered data due to an overall limited flow control window. Future extensions on QUIC (like the proposed datagram one) might change this.
rfc 791 - internet protocol
rfc 793 - transmission control protocol
If routers either ignore the checksum or never add it in, it could work.
Omitting the UDP/IPv4 checksum is permitted because the IPv4 header has its own redundant checksum, but omitting the UDP/IPv6 checksum would leave the IPv6 header with no checksum at all, which is why that's not permitted.
https://en.wikipedia.org/wiki/UDP-Lite lets you checksum the UDP/IP header plus an arbitrary (possibly zero-length) prefix of the payload, but I'm not sure if people actually use that.
HTTP/3 being over UDP will hopefully fix that, while keeping the benefits of HTTP/2.
Even if a packet loss was to occur, it would still have a net benefit over pounding a CDN with countless individual requests.
That was exploited back in the day by download managers which opened multiple connections to the save resource and managed to significantly increase download speed.
Even back then, that wasn't anything to do with multiple TCP connections being fundamentally faster somehow. The speed increase achieved by download managers comes down to:
1. DNS resolution round-robin-ing you to different CDN nodes, such that you're actually making requests to different hardware that each has its own NIC to separately saturate; where if the each server NIC can do 1Mbps, and your link can do 2Mbps, then you need two parallel chunk downloads from separate servers to saturate your link. (This is the same fundamental reason downloading e.g. an OS installer through a torrent is usually faster than downloading it through HTTP from the OS maker's CDN.)
2. Intentional per-connection download-rate throttling applied by many servers (especially those of "download sites" like CNet, which "back in the day" were one of the main places people would experience large downloads); where making multiple concurrent HTTP range-requests to the server at once was a way to bypass the server's throttling. You don't see this advantage any more, as these days the servers of "download sites" have enough smarts to dynamically aggregate downlink quota between nodes in a table keyed by peer IP, such that N concurrent downloads from the same site are always each throttled to 1/Nth the rate.
1. TCP connection increases speed gradually over time until it hit some bandwidth limit. So if you open 10 connections, you'll hit that limit faster.
2. Rare packet drop will drop connection speed and it'll take some time to restore that speed on a given connection. With multiple simultaneous connections and rare drops only one or two connections will experience that speed drop, while other connections might even compensate that with increased speed.
So if your connection is flawless, it does not make sense. Otherwise it might make sense.
This delay is even smaller than RTT due to https://en.m.wikipedia.org/wiki/TCP_congestion_control#Fast_...
0RTT session resumption, while TFO could offer this too most people don't enable it due to broken middleboxes. Header encryption, which hopefully prevents future ossification due to those broken middleboxes. Multiple independent streams within a connection where each stream only suffers from its own packet losses instead of all streams being affected together. This provides a small latency benefit. TCP SACKs and fast retransmits should paper over most of this latency, but not all of it. QUIC also offers more SACK ranges than TCP SACK which might help on links with high, non-bursty packet loss. Then there's path migration, e.g. when switching from mobile to wifi. This could also be achieved with MPTCP, but that's still making its way into the kernel.
I think this notion that you need to work around them, and accommodate, rather than to ignore them, and do things right needs to be thrown out.
All those middleboxes are non-essential, and not are not inherent part of the original design.
I especially don't understand google engineers who decided on TLS 1.3 version hack when the very same situation has repeated itself with 1.3, and many new TLS extensions.
The logic fail for example with IPv6, a giant amount of IPv6 software/hardware is plainly unusable, but this is not a reason to try to hack around it, by trying reimplementing IPv6 in v4.
All of this well explained in this doc: https://http3-explained.haxx.se/
In the end, a QUIC probe, and a subsequent TCP fallback would be worse than well done TCP, and no better than MPTCP/TFO/SCTP/DCCP (all of which were supported by mainline OSes for a long time)
Your only real choices are TCP without any extensions created in the past 20 years or UDP. Some firewalls will block UDP but at least it'll survive the open internet so that's a problem you can get IT to fix.
Also even if we assume that 50% of all internet users have access the most modern protocols as soon as reasonable we still care about how quickly the next half will upgrade.
And in a sense I agree with this vision of the internet. On one hand the internet could be a super optimized distributed communication network with a lot of embedded functionality (I really like the concept of content-centric-networks/name-centric-networks) on the other hand the internet could be a dumb many-ended pipe (the IP protocol)
In terms of how I use the internet in practice I always prefer the dumb pipe model.
This seems like a lot of energy spent to make already fast things just a tiny bit faster. Are there any real world estimates or projections for improvement? Like, how quickly will that Facebook page load or show those Google (ad) search results?
Things feel largely fine, yet here are more "improvements".
And once we have these gains, they will probably be lost in bloated javascript or app layer inefficiencies.
The benefit is not limited to CDNs, any long fat network where we have high throughput and high latency (e.g. most cellular connectivity) should see improvement. I would guess that CDNs may benefit less actually since they tend to terminate near the user and would generally have lower RTT. A typical modern web page has many resources to be fetched, and by making them independent the user experience is likely to be significantly better.
Disclaimer: work at google, not on QUIC but am familiar with the QUIC project and team.
Indeed. I work at Netflix on optimizing cpu efficiency on our Open Connect CDN nodes, largely to reduce power use and capital expenses. We use FreeBSD, ngnix & TCP, and make heavy use of offloads like async sendfile(), TSO, LRO, kTLS and more recently hardware kTLS offload.
Right now, I have a single socket 32c/64t AMD Rome server delivering over 350Gb/s of real Netflix customer traffic. This traffic is all TLS encrypted, and is served across hundreds of thousands of TCP connections.
From measurements we've done, current QUIC would cost about 3x as much as TCP when using software crypto. So my back-of-the-envelope guess is that this box would do about 77Gb/s with QUIC (230Gb/s is the limit when disabling hardware TLS offload and using software crypto).
Are the benefits of QUIC really worth an a 4x increase in the amount of energy required per stream?
Once QUIC has optimizations similar TCP in place, the story will obviously be different. But we're not there yet.
Do you mean to say “YT does JIT transcode” or simply that the huge long-tail corpus means that transcode optimization >>> delivery benefits? :)
Other things to factor in: different mix of mobile users & geographies (incl. access networks), software stacks, hardware costs/goals/footprint, etc.
Whilst QUIC shines when you have a lot of small assets that you want to fetch as quickly as possible?
Like could we have a system where we choose http 2 or 3 depending on the type of data?
The way video is generally served now is actually as a large number of dynamically-selected chunks of the video and associated audio. QUIC makes perfect sense for YouTube/Netflix/Vimeo type VOD, and especially the MPEG-DASH style of streaming.
QUIC would have the advantage that you could maintain that one stream and multiplex both video data and control information over it without the problems you'd encounter doing so on HTTP/1.1 or HTTP/2, but that definitely isn't worth the performance loss you'd get by deploying it today.
I think it makes perfect sense for you to wait for QUIC to be more heavily optimized. Once optimized, I think QUIC has the potential to be even faster, but it isn't there yet for your use case. (For that matter, some of that optimization is work you've put into optimizing existing HTTP and TCP, so it isn't surprising that your existing optimized stack beats current QUIC.)
[1]: https://mailarchive.ietf.org/arch/msg/quic/OnBhC4gCosmlU3VKo...
It seems a bad reason to knock the protocol cos people haven’t implemented something unrelated (hardware acceleration) for it.
I appreciate it obviously would be stupid for you to switch given those stats. But it’s not related to the protocol design or some fundamental shortcoming with QUIC.
But syn/ack is only for data you want to make sure was received. Things like video and audio don't need that to view the content.
So it's one big use is it's a protocol that works over wireless much better for content viewing.
You can test this yourself with a basic server and a mobile phone over WiFi at distance. Over http/2 video will cut out and have a difficult time loading. Over quic it'll just keep going.
QUIC also has a three-way handshake (on new conntections). The equivalent of TCP's SYN, SYN+ACK, ACK, but stacked with the TLS negotiation at the same time.
Also, SYN is only used during this handshake in TCP.
In TCP and UDP, connections are identified using local IP address/local port/remote IP address/remote port tuples. This means, if my IP address changes, (e.g. because I connected to a new WiFi network) all TCP connections that were active before must be abandoned. With QUIC, supposedly, an identifier separate from IP addresses is used, so there are ways how a connection could be resumed. (I'm fuzzy on the details though)
I solved this issue at my last company by just blocking 443/UDP at the firewall so that clients (i.e. Chrome) would fall back to HTTP/2.
When you're building one single thing that's used by 4.6 billion people, it turns out that percentage optimizations matter!
(I work for a company that built a shitty half-baked homegrown QUIC equivalent because in rural Ethiopia, HTTPS handshakes were so slow that they literally just didn't work. Glad that Google is optimizing our percent-of-a-percent use case!)
Unsound hacks that kind of work "acceptable" in A/B test telemetry and slowly break in real life from inherent design deficiency, are almost always worse than something saying from the start "will not work on bugged os/hardware version, but work really well on standard compliant ones"
The TLS 1.3 hack a Google engineer has forced through IETF is now backfiring for example. They did it to hack around a certain brand of middleboxes, but the hack instead broke few other ones, and embedded http servers. They may well errata it, and go back to normal versioning in 1.4, despite putting it on paper in 1.3 that the hack is here permanently.
How is it "backfiring"? It seems to be working for billions of people. If you've got a non-compliant TLS implementation that broke you get to keep both halves, good luck with that.
I'd be interested to know how you feel TCP and QUIC differ in this regard.
It seems to me that they're both reasonably similar protocols that design for slightly different things and make different tradeoffs (reliable single-stream transfer vs reliable multi-stream transfer, ip and port-based endpoints vs abstract connection ids, etc).
I guess you could see the integration of encryption to be an ugly layering violation, but IIUC the primary driver for that decision is less performance and more to prevent ossification by ensuring as much protocol information as possible is inaccessible to middleware.
I'm less enthused about HTTP/3 - I would've liked to see a simpler, "QUIC-ified" version of HTTP/1.1 done first (ie. basically just normal HTTP except with one request per stream), but I guess it's a hard sell to roll back from HTTP/2's efficiencies, and HTTP/2 itself is unsuitable for QUIC since it duplicates much of its functionality.
I also want to see QUIC used for more protocols, especially those that (like HTTP/2) already had some level of stream multiplexing built in (AMQP comes to mind here).
Can someone comment on differences between the two patches?
There isn't huge difference between them from technical point of view. But it's your decision what of them you will use.
They have make WireGuard implementation in Rust. Here is original WG-RS: https://git.zx2c4.com/wireguard-rs/about/
But CF is using their own implementation for WARP: https://blog.cloudflare.com/1111-warp-better-vpn/ "We built WARP around WireGuard, a modern, efficient VPN protocol that is much more efficient than legacy VPN protocols."
Here is other repo about projects written in Rust: https://github.com/cloudflare?q=&type=&language=rust
Besides that
- one of the implementations has unit-tests
- one of the implementations is in written in a by-default memory safe programming language
- one of the implementations might become available to Nginx users without having to invest extra work
Take a Remote Desktop protocol used in Microsoft Windows. It can work over TCP but recent revisions tend to automatically switch to UDP. And know what? They are not reliable to the point that customers have to turn the UDP layer off. TCP gives a slightly worse latency but it is much more reliable and thus usable. Thanks there is a Group Policy for that.
I'm not even talking about Google as a company who constantly tries to attack the network infrastructure with its variant of EEE (Embrace, Extend and Extinguish). What's the end game? The crippled protocols worldwide imposed by the ad casino company? No, thank you. Internet must remain free of all of that.
I guess they didn't implement reliability then. This has nothing to do with QUIC.
> Embrace, Extend and Extinguish
Yeah, working on an IETF standard is EEE now? LOL
Seems to be working fine for hoi polloi. For now, at least.