Comparing TCP and QUIC
potaroo.net
potaroo.net
Do the benefits of QUIC really justify the economic and environmental impacts of that kind of loss of inefficiency on the server side?
And yes, I know that some of these offloads are being worked on, but they are not here today.
“Takeaway: We observe that QUIC provides significant improvements over TLS/TCP in low-bandwidth and high-RTT regions for video downloads. QUIC handshakes towards YouTube media servers offer an improvement of 534 ms (IN) and 406 ms (DE) when compared with TLS/TCP. We also observe that the overall download rate for TLS/TCP is higher than QUIC partly due to kernel optimizations such as LRO available for the TCP stack. QUIC provides a better video streaming experience with a lesser number and duration of stall events compared to TLS/TCP. We observe that TLS/TCP exhibits up to 50% longer stall durations compared to QUIC at 50th percentile for high loss networks.“
https://vaibhavbajpai.com/documents/papers/proceedings/quic-...
Also the study discussed about the server-side CPU usage. If you are worried about server-side offloading, you can still stick to TCP/IP until the hardware acceleration and other offloading catches up.
The other advantage is that thanks to being based on UDP, you get a STUNable version of TCP, which is really great for p2p apps like Syncthing.
For making sense in the data center, or for delivering bulk data, QUIC just doesn't have a killer feature really, compared to the ages old battle-tested TCP.
> There may be some performance penalty of shifting the transport code from the kernel to user space
This makes it sound like a kernel has potential for slightly more optimised implementation. But I think it's more than that - the transport code can be completely offloaded from the CPU to the network card/processor. That can only happen if it's abstracted behind syscalls, not written in user space.
If QUIC sees wide adoption then it's just a matter of time before the protocol starts moving up stack.
I don’t know enough about it to know if that’s aspirational or an active project. But the idea of being able to push code to the “right” side of bottlenecks is a running theme, including in questions of user space versus kernel space. Chatty communication kills throughout and/or latency.
https://www.netronome.com/products/agilio-cx/
I've been meaning to get my hand on one of these to play around with eBPF offload.
There is also XDP (a subsystem of eBPF) which lets you take action before the packets run up the entire network stack.
Netronome doesn’t print their prices. That’s not a great sign. If you have to ask, you can’t afford them.
Throughput-wise, I've found in real-world testing that QUIC is often slower than TCP: (1) QUIC uses more CPU, due to the processing in user-space. 1Gbps required 3 CPU cores. On my 1x-CPU VPS, QUIC maxed out at 400Mbps due to the CPU. With TCP+TLS, I could comfortably achieve 5Gbps. (2) QUIC was less resilient to packet loss (surprisingly). This was particularly noticeable on mobile devices.
If your use-case is to move bytes between powerful servers over a reliable, wired connection, QUIC may beat TCP in most ways that matter. But for use in real-world mobile apps, TCP may still offer better throughput.
Caveat: This is all data from using the quic-go package. The C libraries may well be more efficient :)
That’s not inherent to the QUIC protocol itself though, that’s an implementation decision. There is no fundamental reason why QUIC couldn’t be implemented in kernel-space, such an implementation would still be conforming.
TCP processing can be also offloaded to network cards today, so the same might need to happen for QUIC to be similarly efficient.
QUIC is optimized for Google's use case - Google client talking to Google servers, with many Google streams combined into one big pipe. This is not the normal non-Google case. Early performance numbers from Google only indicated a relatively small gain (15%?) even for that case.
If you're loading several images over the same HTTP2/TCP connection and you miss a packet of one then you have to get later packets resent for all of them. Even if the delayed packet does eventually arrive without needing to be resent, it still has the effect of holding up all the other streams. With HTTP3/QUIC, a missed or delayed packet does still affect the stream it's from within that connection (still a pity) but at least the other streams carry on unaffected.
This problem in TCP is called head-of-line blocking:
Only if you're multiplexing multiple streams inside one TCP connection. Multiple TCP connections will not affect each other.
Browsers typically have multiple HTTP connections open as a compromise so that you don't need a new TCP connection for every file but don't have the head of line problem so severely. But it's not as good as fixing the underlying problem.
The main use case I need is to have reliable streams and unreliable datagrams that continue working even if either peer's IP address changes. Something like that would allow cell phones to form scalable p2p mesh networks, for example.
It needs to be encrypted and punch through NAT in a fully automated way, falling back to a (secure/anonymous) matching server if both peers are behind NAT. I don't know about that last part, but it appears that WebTransport over HTTP/3 over QUIC over UDP might be able to do most of that and be a potential replacement for WebRTC data channels:
But for those games that do use QUIC and do have private servers, hacks will be needed to keep the games going long term (20+ years). Retro-gaming is going to be much harder in the future.
WebRTC has a really cool feature called `ICE Restart` that solves the 'walk out the door' problem.
But ya, I got windowing working, and came up with a scheme to store the app's identifier (4 digit creator code on macOS) in a checksum by subtracting the actual checksum to be left with a remainder like 'app1'. So the game would transmit on checksum app1 instead of port 12345 for example. But I realized the overhead of computing the checksum might make it vulnerable to denial of service attacks, so maybe a separate field might have been better for faster filtering. I halfway implemented some slow start and Nagle algorithm stuff too. But NAT negotiation with STUN/TURN/UPnP/NAT-PMP or similar killed the project. That's so hard to solve for humans that I wouldn't even try now, I would probably enter all of the states in a giant truth table or graph and write a stateless solution with the time element removed instead.
This was over 15 years ago though so the C++ code isn't really any good anymore. Also because of buffer overrun exploits in C, I think it's better to use a proven (formally tested?) framework like WebRTC instead, even for low-level stuff like games. We have GHz processors now that can handle any level of complexity around ~1500 byte packets and still saturate pipes, so trying to go bare-metal there strikes me as premature optimization that isn't usually worth the risk it exposes.
The next thing that got me was, I didn't know about coroutines. So I had incredibly elaborate state machines for the game logic that quickly became unmaintainable even for the simple puzzle games I was prototyping. Today I would use a state manager like Redux or a software-transactional memory (STM) or a conflict-free replicated data type (CRDT), maybe distributed with Raft/Paxos somehow so that all peers can see the same state. An event-driven interface like the one used by Firebase/RethinkDB/Supabase, maybe CouchDB or similar, would be good too. I got stuck on solving consensus, like when 2 people hit each other at the same time, but maybe that's been solved.
And then there's the matchmaking/lounge server. I cobbled something together using an IRC-style syntax, but was never happy with it. Maybe something like Matrix would work there.
Believe it or not, I haven't actually used WebRTC yet, but want to. I also don't want a competing standard in WebTransport. I think what I was always looking for was a connectionless reliable stream, which was never a thing on the web, which set innovation back perhaps 25 years because only a handful of apps like Skype managed to pull it off. It appears that QUIC comes closest. And if WebRTC is the only one that's solved NAT and the connectionless part with ICE Restart and peer identifiers, then I'll consider that the standard.
<rant>
Sorry this got a little long, I'm just passionate about the injustice of stuff like NAT. I feel that the lack of solutions around this stuff was deliberate by the telecom industry. It all but ruined my game development career by making it difficult to compete with large AAA studios after the mid-2000s (until the arrival of Unreal/Unity perhaps). This is 1 of 100 rabbit holes that I've been down only to fail. So I appreciate that people like you are shedding light on this stuff so that maybe the next generation will be able to work feely without artificial barriers holding them back.
Ignoring this extremely dangerous outcome of a QUIC only world, this write-up is excellent and really clears things up.
I know quinn (a rust QUIC library) definitely does, though you have to enable custom verification as a compile option (which is reasonable).
For proprietary software it's idiomatic to use a private CA for securing connections to a central server, to avoid the risk of DNS hijacking or a malicious third-party CA. For open-source software it will, definitionally, allow substituting different server key validation options for users who prefer to run their own servers.
The only real problem QUIC's mandatory TLS causes is that localhost development becomes marginally more difficult, but even as someone who loves running my in-development stuff on localhost it's clear that the security of the internet as a whole is of far greater value.
Private CAs solve other problems for internal uses but not this big important one for the public web.
By the time the major browsers drop support for plaintext HTTP, that protocol will be as quaint and archaic as Gopher is today.
(and we'll all be better off for it -- good riddance to anything ASCII-based or unauthenticated)
[0] https://blog.mozilla.org/security/2021/07/20/stopping-ftp-su...
HTTP/1.1 will be obsolete when intranets and tiny devices are obsolete. Well, it looks like that could become true...
That’s not comparing apples and oranges-HTTPS was originally just (and often still is just) plaintext HTTP/1.x wrapped in TLS. Whereas, SFTP and SSH are not FTP/TELNET over TLS, they are completely separate protocols. FTPS and TELNETS are the original protocols over TLS, but have seen only limited adoption (mostly in mainframe environments, in which SSH and SFTP perform poorly.)
Even HTTP 2 and 3 are just a binary encoding of the HTTP/1.x protocol over an encrypted transport. By contrast, SSH and SFTP are essentially unrelated protocols to TELNET and FTP, as opposed to just binary encodings of them. (The TELNET protocol is already non-text-based anyway-the data may be mostly text but the control messages have never been.)
I'd be more optimistic if not for the (cynical?) fear that those most in need of encryption will be the first to be denied it by the force of the state, a day only accelerated by its increasing ubiquity. Your government can force you to install a state-sanctioned CA cert, almost entirely undermining TLS from that perspective, which Google can/will do nothing to resist, and their parallel desire to deprecate plaintext HTTP is entirely orthogonal to this.
Google has been doing a great job of killing the "personal internet" by making it undiscoverable for several years, but now that we require a third party to somehow be involved, and likely a fee of some sort to be paid- I don't trust that the likes of Lets Encrypt and its duplicators to be available forever, the original "anyone can publish" dream of the web is dying in front of us. This: https://justinjackson.ca/words.html does not require SSL in any form, in fact most websites don't need it at all- and the obsession with even blogs wanting you to make accounts and to have to login, its all just heading in a very wrong direction.
Its just deplorable to me that I now need a third party to essentially tell me that I can be on the Internet. Its a huge step backwards in so many ways. And no one seems to care!
> Its just deplorable to me that I now need a third party
You've always needed one or more third parties (ISPs, backbone, ICANN, etc), this is no different.
It does if they install the CA cert.
UI is not pretty, but not insurmountable on standard browsers, OSs, or phones, and there is no interference unless you are working on a domain-joined Windows PC or your phone is under some sort of MDM.
These UI issues with this process could be resolved by some browser UI work by parties that aren't interested in perpetuating the current status quo and can get past handholding boomers apparently willing to enter bank credentials into anything that moves.
Or just use Gemini.
4.2 Server certificate validation
Clients can validate TLS connections however they like (including not at all) but the strongly RECOMMENDED approach is to implement a lightweight "TOFU" certificate-pinning system which treats self-signed certificates as first- class citizens. This greatly reduces TLS overhead on the network (only one cert needs to be sent, not a whole chain) and lowers the barrier to entry for setting up a Gemini site (no need to pay a CA or setup a Let's Encrypt cron job, just make a cert and go).
The problem is that while you can still add root CAs on modern Android, apps have to specifically opt-in to user-added root CAs. They can ignore them as they wish and I believe they have to even opt in to third-party CAs. Needless to say most apps don't bother. This is a big problem with self-signed certs on Android. I believe this was changed in version 7 or so.
Not sure how this works on iOS but on Android it's really a problem.
Gemini is fine. I love trust on first use for TLS. But there's no reason to drop HTTP/1.1.