QUIC and the end of TCP sockets
codemia.io
codemia.io
Many of the claims are dubious. TCP has "no notion of multiple steams"? What are two sockets, then? What is poll(2)? The onus is on QUIC to explain why it’s better for the application to multiplex the socket than for the kernel to multiplex the device. AFAICT that question is assumed away in a deluge of words.
If the author thinks it’s the "end of TCP sockets", show us the research, the published papers and meticulous detail. Then tell me again why I should eschew the services of TCP and absorb its complexity into my application.
In the mean time we could have had nice things: https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...
SCTP would be a fantastic protocol for HTTP/HTTPS. Pipelining, multi homing, multi streaming, oh my.
As in, we wouldn't have to implement it on top of UDP
I’m not sure I agree though, because many firewalls already pass other protocols today, like GRE, IPSEC, etc.
SCTP is another high-performance protocol that provides many of the same network and performance features as QUIC, but it has been around for ~25 years. It was widely implemented. It was even in Linux 2.4 and available for Windows XP and MacOS. WebRTC in browsers can use SCTP, which makes sense (it predates QUIC) but is actually a bit of a problem if you want wide interoperability with WebRTC.
But despite SCTP being around and widely implemented, and doing many of the good things QUIC does, SCTP was never widely adopted, nor seriously considered for HTTP, mainly because of the problem of firewalls. (And to be fair, some operating system API limitations).
Basically, if TCP were invented today, it could not be deployed succesfully over much of the internet unless it was wrapped in UDP and encrypted.
UDP-with-encryption is now playing the role that the IP protocol is supposed to play, just because too many firewalls block nearly every IP packet that doesn't have type TCP or UDP. If UDP is used but without encryption for a popular protocol, eventually too many firewalls deep-packet-inspect that UDP protocol too, and do the same kinds of blocking, tampering or making protocols break.
This sad problem is called protocol ossification. As Google's AI puts it: "QUIC is the first IETF transport protocol to deliberately minimise its wire image to avoid ossification.". See also https://en.wikipedia.org/wiki/Protocol_ossification and https://http3-explained.haxx.se/en/why-quic/why-ossification
There are some downsides to QUIC's ossification resistance. I did some work for a mobile equipment manufacter who adaptively manage the queues for multiple TCP streams from different users and different applications from the same user, to help ensure a better, fairer and lower-latency experience. QUICs properties made it difficult to measure what's going on at the protocol level, forcing the queue management to use crude statistical estimators instead, which don't work well on bursty protocols.
Going back to David Reed, this is specifically why UDP exists: as the extension interface to build more non-TCP transport protocols.
On top of that, I believe the UDP checksum can be omitted as well at least on some OSes (and is arguably not necessary for fully encrypted/authenticated payloads) – leaving really just the two bytes for the "length field".
If we just allowed all L4 protocol numbers through and ditched NAT we could have nice things. Or we could kick it up two layers to use QUIC for what SCTP could have been.
Having a diversity of IP protocols isn't a nice thing. The designers of TCP/IP made a protocol specifically for doing the thing you wanted to see SCTP do: it was called UDP.
we're fighting for bits here!!! if you want to be 100% safe you need 576 packet size, UDP header is 1.4% of that
I am very aware of what you can do with UDP, I have done some very fun work trying to minimize bandwidth usage on crappy mobile connections by using and abusing it. But I think at the end of the day it is an engineering crutch.
If we insisted on properly supporting a diverse set of L4 protocols years ago we wouldn’t have wound up with NAT and slow adoption of IPv6. Address exhaustion would have been a real pressing issue. Instead you can’t even ping a server behind a NAT and firewalls run out of memory trying to manage stateful connections.
UDP is a pretty elegant design for what it is but it is barely good enough to allow us some room to make things work. Ultimately it did limit us more than it enabled us.
And I agree: it stifled what could have been a much nicer to work with set of protocols and who knows what could have been created had we not just said "well there is always UDP if you want to do your own thing".
The fact of ICMP not being itself a UDP protocol caused major problems for systems programmers, because it meant that OS kernels all "owned" ICMP, provided only a baroque sockopt programming interface to like 5% if it, and required userland programs to hold suser privileges to do any real ICMP work. Awful design. And ICMP is slow-pathed by routers, because it isn't UDP.
UDP literally doesn't do anything but multiplex raw IP. Unless you're actually worried about the 8 bytes the header takes up, there's no reason, none at all, to slide a new IP protocol anywhere but on top of UDP. Again: that's why UDP was designed in the first place. You can go look this up! David Reed still talks about it!
PMTUD breaks when ICMP is blocked.
The same argument can be made that everything but HTTP being blocked is not a problem because everything can be transported on top of HTTP.
Imagine at the beginning of a connection, sending a burst of packets, you could send (ignoring tcp timestamps because it makes the math hard, PAWS is a waste of bytes for most flows, and etc) [0,1460); [1400, 2800), ...
If you get an ack of the first packet, great. If not, you resend it as a 1400 byte payload and probe again in a future burst. Maybe even premptively resend the first packet as a 1400 byte segment after a short delay. Anyway, have enough failed large packets and probe smaller. Probe bigger again every so often if the connection stays open for a meaningful amount of time.
WebRTC has been doing just that for peer to peer data connections (which need TCP-like semantics while traversing NATs via UDP hole punching).
UDP is a good solution but all it does is provide an 8 byte overhead and nothing that IP itself doesn’t provide for something like SCTP.
Source and destination port just seem like a reasonable baseline for alternate protocols, that's 4 more – leaving just the checksum. (If you're really desperate for space and have integrity provided by your protocol, you can even cram two more bytes in there!)
Sure, it would be conceptually nice to be able to skip UDP, but I think in terms of performance it absolutely does not matter.
But my point isn’t even about performance. It is about the fact that NAT and IPv4 address exhaustion and bad firewall practices have killed any innovation of level 4 protocols. Imagine if instead of TCP, SCTP had won the protocol wars in 1980-1990s. Or even better if we had realized that we were going to run out of IPv4 addresses much earlier when the cost of switching was smaller. It would have been so much better to have firewalls that don’t filter anything but protocols 6 and 17. We could have had the opportunity to experiment with different types of transports, baked encryption in at a lower level, etc.
Basically where we are is that we have 6 and 8 dot LEGO bricks to play with and are told that we can build anything with those but aren’t allowed to play with any other shapes.
Does it not? Not sure if it's really mandatory, but I believe one rationale for IPv6 getting rid of both its checksum and length fields was that both TCP and UDP duplicate both fields.
Given that QUIC doesn't have its own length field, I would imagine it relies on that of UDP, at least in situations where the lower layer does not provide reliable framing?
> Imagine if instead of TCP, SCTP had won the protocol wars in 1980-1990s. [...] We could have had the opportunity to experiment with different types of transports, baked encryption in at a lower level, etc.
Would we? Instead of TCP, SCTP would have become ossified in the stack all other things being equal (in particular the need for something like NAT due to IPv6 exhaustion), no?
But even UDP is heavily restricted in most cases.
Too many things that people actually need to work run over it these days, including VPNs, videoconferencing etc.
Good news, then: QUIC uses UDP.
> As for as QUIC support? Never gonna happen here.
So generic UDP is allowed, but QUIC is specifically detected and filtered? That would be bizarre.
And UDP hole punching is a crutch for the fact that we still use IPv4.
At the time HTTP2.0's multiplexing was known to be bad for anything other than perfect, low latency networks. I hope this was because people had faith in better connectivity, rather than ignorance of how mobile and non-lan traffic worked.
You should probably at least try QUIC now, but you can get past HOL blocking by having multiple TCP streams. Its super cheap, cheaper than QUIC.
And also super inefficient, since it duplicates the TLS handshake across streams and uses more resources in the OS and middleboxes (like them or hate them, they're a thing that might throttle you if you go too crazy with connection-level parallelism).
That's on top of very poor fairness at bottlenecks (which is per TCP stream unless there's separate traffic policing).
Depends on your usecase of course. if its something thats only around for a minute, then yeah, but over an hour, makes no difference at all.
QUIC eats more CPU per MB of data than plain TCP/TLS. but that might be solved by more modern offload/kernel optimisations
In any case, I'm optimistic that QUIC has a bright future. I don't expect it to replace TCP, but give us another tool we can use when it is called for.
Is that not the case for TCP as well? Most congestion control algorithms just assign new meanings to existing wire-level flags (e.g. duplicate ACKs), or even only change sender-side behavior.
> QUIC gives* control back to application developers to tailor congestion control to their use case*
That's what it actually does: It moves the congestion control implementation from the OS to user space.
In that sense, it's the same tradeoff as containers vs. regular binaries linking to shared libraries: Great if your applications are updated more often than your OS; not so great if it's the other way around.
If I understood modern application development correctly, this interprets as "The developers will import another library which they don't understand and will wreak havoc on other applications' data streams by only optimizing stuff for themselves".
Again, if I remember correctly, an OS is "the layer which manages the sharing of limited resources among many processes which requests/needs it", and the OS can do system-wide, per socket congestion control without any effort because of the vantage point it has over networking layer.
Assuming that every application will do congestion control correctly while not choking everyone else even unintentionally with user space's limited visibility is absurd at worst, and wishful thinking at best.
The whole ordeal is direct violation with application separation coming with protected mode.
Why? All practically used variants of TCP achieve fairness at the bottleneck without any central arbiter or explicit view of the upstream congestion situation. Besides, UDP has never had congestion control and has been around for decades.
> The whole ordeal is direct violation with application separation coming with protected mode. [...] The whole ordeal is direct violation with application separation coming with protected mode.
Are you concerned about fairness between applications on the same OS or fairness at the bottleneck, usually upstream from both application and OS/host?
In the former case, the OS always gets the last word, whether you're using TCP or your own homebrew congestion control via UDP, and you can make sure that each application gets the exact same fair egress rate if required. In the latter case, nothing prevents anyone from patching their kernel and running "unfair TCP" today.
I think the hope is that OS developers will continue to generally pursue (tcp) congestion control that is at least reasonably fair. But if it's under the control of app developers, there's less of a track record of doing the right thing.
Congestion management at bottlenecks is very hard at the bottleneck. If a 40gbps port is congested, the routers on that port almost certainly do not have the ability to do any sort of fair queue management. So everyone has to play nice.
It is not terribly hard for an app developer to get millions of installs; it's a lot harder for an app developer to get millions of patched kernels.
That said, if the whole deal with quic is google wanted userspace congestion control, which is understandable, I don't see why they didn't just do that. It's not like they don't control a large amount of servers with a ton of traffic, and they also control the OS (to some degree) on a large amount of clients. http/2 multiplexing would still be as obviously flawed as it was on release. You'd still have head of line blocking on http/2, but with the right settings, you can just have multiple tcp connections... TLS 1.3 0-RTT and maybe tcp fast open could get round trip counts reduced when appropriate. With proper congestion control, you could share the congestion context for multiple connectjons tk the same peer. QUIC would still have the upper hand with the ability to selectively not retry on unacknowledged data, but I don't know how often that's really used.
I stopped reading here. This isn’t really an essential property of QUIC, there’s a lot of good reasons to eventually try to implement this in the kernel.
Most OSes don't let you send raw TCP segments without superuser privileges, so you can't just bring your own TCP congestion control algorithm in the userspace, unless you also wrap your custom TCP segments in UDP.
Drown other applications for your own benefit?
How would you swap out the TCP congestion control algorithm in an OS, or even hardware, you don't control?
> Drown other applications for your own benefit?
Fairness equivalent to classic TCP is a design goal of practically all alternative algorithms, so I'm not sure what you're implying.
It's entirely possible to improve responsivity without compromising on fairness, as e.g. BBR has shown.
On the contrary, introducing their own novel/mysterious/poorly implemented congestion control algorithms is not a thing I want userspace applications doing.
And if you worry about hostile applications on your own hardware, the OS is an excellent point to limit what they can do – including overwhelming your network interface.
One might even call this "congestion control"!
While you're still on your own host, your OS has complete visibility into which applications are currently intending to send data and can schedule them much more directly.
> Why QUIC’s user-space transport lets us ‘kill’ the old app-level event loop
But then doesn't seem to mention that topic ever again. I don't see how QUIC changes that much.
Is that really a QUIC thing?
QUIC implementations do not allow for anyone to connect to anyone else. Instead, because it was built entirely with corporate for-profit uses cases in mind and open-washed through the IETF, the idea of a third party coporation having to authenticate the identity of all connections is baked in. And 99.999% of QUIC libs, and the way they're shipped in clients, cannot even connect to a server without a third party corp first saying they know the end point and allow it. Fine for corporate/profit use cases where security of the monetary transactions is all that matters. Very much less fine for human uses cases where it forces centralization and easy control by our rapidly enshittifying authoritarian governments. QUIC is the antithesis to the concept of the internet and it's robustness and routing around damage.
Alas alas, basically stalled out, afaik no implementation. I wish Microsoft (the spec author) or someone would pick this back up.
It just uses self-signed certificates, which is maybe conceptually slightly clunky compared to "pure" anonymous TOFU, but allows reusing existing stacks.
Even if you don't want to get a Letsencrypt certificate, you can always use a self-signed one and configure your clients to trust it on first use or entirely ignore it.
SSH also uses "mandatory host keys", if you think about it. It's really not a question of the protocols but rather of common client libraries and tooling.