Can QUIC match TCP’s computational efficiency?
fastly.com
fastly.com
Danger, non-server hardware detected.
we limited the CPU’s clock from 2.2GHz down to 400MHz
Danger danger, system imbalance approaching critical levels; please evacuate to a safe distance.
It shouldn't be taken for granted that a CPU bottleneck exists, although from Netflix's work we know that serving >100G from one server is CPU-intensive.
Sure, there's the rare case of a connection moving to a different L3/4 flow identifier causing a cache miss, but that's not something that happens often enough to be a relevant bottleneck.
RFS flow tables are installed by sendmsg and recvmsg, so I fail to see why RFS doesn't benefit UDP just as much as TCP. If you are using RFS, your QUIC packets should still arrive on the correct CPU.
Doesn't that mean that you should have more NICs?
From my recollection, the main thing that's made a computer a server or mainframe for the last 50 years has been that such systems have enough IO channels to saturate the CPU (i.e. they're intended to be CPU-bound, not IO-bound, under heavily-concurrent workloads.) If you take all the IO-offload cards out of a mainframe, it's just a regular computer.
For example, this Cascade Lake Xeon has 48 lanes of PCIe gen 3: https://ark.intel.com/content/www/us/en/ark/compare.html?pro...
If you give yourself 32 of 48 lanes for networking, that's only 256 gbits of PCIe bandwidth. So each 16x slot is limited to a 100 gbit card.
And that's a 22 core processor. With DPDK, coming up with a application that could process 200 gbit is almost trivial.
While the talk is centered around RISC-V, there's nothing RISC-V specific about this, just that it enables whimpy cores to control multi-Gbps traffic.
[1]: https://www.youtube.com/watch?v=LDOlqgUZtHE (Accelerating Computational Storage Over NVMe with RISC V)
The amount of TCP hardware offload in linux right now is actually fairly limited. Kernel developers have largely rejected true in-hardware TCP stacks for a while now (although you can get this functionality with high end NICs and custom kernel builds). That said, since QUIC generally reduces the ack frequency it should be possible to match or exceed TCP with hardware offload. Especially when you look at total throughput with many streams open.
TCP is baked into firmware on edge routers and firewalls and wifi devices that are beyond the period of support and the companies that created them aren't interested in supporting them or have gone out of business.
There are other protocols in layer 4 that could have been used that are better on some workloads than TCP, but TCP and UDP were the only things configured/supported; many firewall default configurations filter out things they don't recognize so anything new is a non-starter. And because of the perception of responsibility when it comes to networks, if I support a new protocol and you can't connect to it because there's older devices between you and me, it becomes my problem.
Anyway, anothing not 'normal' is being filtered out somewhere. We even (at one startup) considered tunneling RTP through HTTP! My god.
> One concern about the move from TCP to UDP is that TCP is widely adopted and many of the "middle-boxes" in the internet infrastructure are tuned for TCP and rate-limit or even block UDP. Google carried out a number of exploratory experiments to characterize this and found that only a small number of connections were blocked in this manner.[3] This led to the use of a rapid fallback-to-TCP system; Chromium's network stack opens both a QUIC and traditional TCP connection at the same time, which allows it to fallback with zero latency.[17]
QUIC can be 'a bit better' in 'some scenarios', but most of the time it is totally irrelevant, except for the fact that it requires TLS 1.3 which is a good requirement overall.
1: https://www.cs.cmu.edu/~dga/papers/incast-sigcomm2009.pdf
I'm not sure your conclusion can be drawn from the cited paper alone, which is now 11 years old. The paper describes a phenomenon that occurs when switch hardware buffers overflow on high-throughput networks -- a problem that has since been resolved in many datacenters that use software-defined networking and more modern hardware.
Keep in mind that there are only two outcomes to networking: the frame is either forwarded or dropped. Whenever you encounter software-defined networks you need to ask which frames the network is dropping, because that's the only action an SDN can take.
Incast is a stochastic phenomenon and the point of the paper is there is some probability, which is not zero, that too many frames will arrive at the same point in the network at the same time, forcing some to be dropped. This probability cannot be zero regardless of the buffer depth. Eric Dumazet has written a lot about how adding entropy in packet scheduling at the host level helps avoid frame drops. The fact is the problem cannot be solved by the network, it has to be solved by the hosts at the edge (and this is basic nethead-vs-bellhead philosophy going back 40 years).
Generally, when we saw losses, we were able to work with the datacenter to track down bad cables, or overloaded paths, or overloaded fabrics (which was usually in their 'legacy' datacenter). I expect the datacenter networking was overprovisioned (separate backend network, dual NICs on hosts, multiple uplinks on switches, etc), but it was darn reliable. Of course, it would have been nice if the people who owned the datacenter would have been monitoring all the stuff their customer (us) was able to find was broken. And, some sort of multistream connectivity would have helped avoid the paths with loss; ocassionaly the loss would be bad enough to impact service, and we'd need to force reconnects until the tcp stream went on a different path.
By saying it's "inappropriate for datacenters," I think he painted too broad a brush - obviously not all are the same, and not all run at Google scale. Depends on the datacenter and the workload.
I've been hearing arguments that "TCP sucks and is irredeemable" for decades, too, yet the world still turns and our devices that use it continue to work reasonably well at increasing bandwidth (2.4kbps to 40Gbps+) - at least as far as the public is concerned.
https://www.ietf.org/proceedings/97/slides/slides-97-tcpm-tc...
I'm trying to find the actual general benefit, but so far I'm not finding it. There is a technical benefit, but not on the scale of say h2 multiplexing or binary vs. plain-text protocols.
Of course, you could just buy a TLS accelerator if you wanted one.
Another way is if it takes a couple CPUs to run my application, a CPU or two to drive the storage, and a CPU to do TLS, why would I also spend a CPU on TCP if my network card already has one that does that?
That's a little bit like what programmers are talking about when they are talking about "multi-core programming", except those are general-purpose "cores", and (presumably) these "cores" can only handle doing TCP.
I'm not sure about the implementation details of GSO and TSO in Linux. But I know a thing or two about the hardware: the Intel Niantic/82599 NICs support UDP transmit segmentation offloading, I've checked the datasheet real quick and it says "Note that current UDP segmentation offload is not supported by any standard OS."
Not sure if this is just an outdated comment in a 10 year old datasheet or if there's some performance to be gained by using this UDP segmentation offload for quick.
Edit: Should have clicked on the link in the article [1] before posting here, UDP segementation offloading is coming to lots of NIC drivers in 5.5.
Always interesting how the hardware often supports way more features than the drivers expose and you suddenly get new hardware features on a 10+ year old NIC model :) (Similar story with IPsec offloading on the Niantic/82599)
It's also supported on the old chipset from the motherboard I bought more than half a decade ago, and every chipset I've used in the server space in as long as I can remember.
A phrase like "cute but completely incomparable" doesn't teach the reader anything. It would be much better to say what the relevant aspects of "transmission intelligence in the Linux TCP stack" are, and what makes them relevant.
A 747 is not comparable to a Cessna. The later is cute but not comparable and it doesn't require a dissertation to comment on the quality among other computing professionals.
"A 747 is not comparable to a Cessna" doesn't help because it's really still just calling names. What would help is an explanation of why you're saying that. What are the significant differences and why do they matter? Lots of readers here would find that interesting. If you don't want to spend your lunch break writing an explanation to help people learn, that's totally fine. But at least please don't post empty comments that make HN worse and break the rules. Not posting is always an option.
You may be overestimating the extent to which readers here will already understand the details of what you mean. No doubt some do, but I'm pretty sure the majority don't. Even most computing professionals aren't specialists in this area. That's why the opportunity to share information that readers find interesting and can learn from is such a good one. You can certainly (indeed, you should) make a strong critique of inflated and misleading claims, but do it with solid information, not shallow putdowns that any internet snarker can write whether they know anything or not.
Wow, that's quite a strong sentiment, especially given CDNs are your area of expertise [0].
I was (am?) of the opinion that Fastly is a genuine competitor to Cloudflare given their product offering, and that their engineering was more than a match (though, to be honest, I'm mostly going by the presentations their engineers have given over the years). Your statement, to me, makes it seem like Fastly is smokes and mirrors. I'm honestly surprised.
Reno can work nicely for low packet loss links, though.
So basically this isn't a competent test; it isn't the production grade QUIC from google, it's a toy and the result is complete nonsense.
I'm no TCP (or even congestion control) expert, but I wouldn't have needed to test their code in real life to tell exponentially reducing send window on a lost packet is just not going work in real life in 2020.
They should have at least tested their code with simulated jittery latencies and packet loss typical for modern 4G networks. I bet Linux TCP would have left their implementation in dust.
Another thing that caught my eye was their assumption that MTU is same on return path... probably often true, but definitely wouldn't count on it!
Huge respect for them for putting their necks out and releasing code! Maybe this is their intention as well, to gain understanding through criticism.
Also, perhaps this was just their MVP? It would certainly explain weird choice for congestion control.
[0]: https://github.com/h2o/quicly/blob/master/lib/cc-reno.c
You could get very close to maximum possible just by massively delaying & coalescing ACKs. In other words nearing blind transmission without caring about packet loss at all.
But latency could get pretty bad!
For HTTP/3, then, would fq-codel + BBR work nicely for foreground traffic like voice or video over WebRTC and, say, LEDBAT for background traffic like HTTP downloads or, say, bit-torrent over WebRTC?
(I realise congestion control is long from being solvable: https://news.ycombinator.com/item?id=17455276)
I meant stream of the receiving party would effectively block on any lost packet. You can't read past what you don't have yet.
Imagine a stream that consists of 10 packets, and for simplicity that each packet takes a second to send. (Real life scenario would be about several orders of magnitude more packets and similarly shorter packet transmission times.)
You receive packets 1 and 3-10. Thus in one second the application can see data from packet 1. But application's read is going to block until you also receive packet 2. Reception of packet 2 can't happen until 8s is spent receiving packets 3-10 and additional 1s resending 2 (plus whatever it took to send the ACK).
So data from packet 2 would be 9s late from receiving application's point of view.
Bulk transfers would approach maximal possible transfer rate because sender lost no time waiting for ACKs, but this would be pretty bad for something like remote desktop or streaming video.
No sane general purpose network stream protocol would implement this policy for obvious reasons. But you'd sure get high bandwidth for file transfers!
No, and that's a current drawback of QUIC. But of course it's just a current drawback - the investment cost of integrating this into the operating system vice applications will certainly be eclipsed by the benefits as QUIC is deployed more throughout the internet
I feel like Google in particular loves to do that stuff because while they control the internet and the browser, they don't (yet) control the OS so they have a very strong incentive to push stuff up in order to retain control. They don't have to ask anybody to add things into Chrome, they have to get third parties on board to extend the kernels (outside of Android at least).
Things they could do:
a) enable MTU blackhole probing, so people behind dumb networks can talk to servers that don't artificially reduce MSS. Apple has a pretty agressive implementation of MTU probing, and it's very effective.
b) put in hooks for userspace congestion control. Make sending faster (or not) with updates from Google Play services (or whatever)
c) the article mentions sending fewer acks. That could probably be done in TCP too --- it makes sense to make that a setting that they could roll out.
d) some sort of name and shame program for bad networks / bad network devices. It's 2020 and people running PPPoE and sending MSS 1480 when it's really 1472 should be ashamed of themselves, but big companies just set their servers to MSS 1460 and call it a day. :(
QUIC is a standard, it will be stabilised in time.
TCP can't be improved on much at this point, and there are some pretty massive flaws in it that QUIC solves.
TCP isn't going away any time soon, or being "replaced" by QUIC, but for certain applications QUIC is going to be a massive improvement, albeit and an increased complexity and application size cost.
There are over 10 implementations of QUIC at this point in time, you don't have to go with Google's approach, https://en.wikipedia.org/wiki/QUIC#Source_code.
Everyone involved in making these protocols (HTTPS/WebSocket/HTTP2/HTTP3/etc.) are just victims of:
“It is difficult to get a man to understand something, when his salary depends upon his not understanding it!” - Upton Sinclair
It also allows ISP shenanigans like content modification to add unwanted advertising and unlimited tracking, even spying.
So could you please explain your reasoning?
HTTP/2 has the TCP head of line problem which makes it completely useless.
HTTP/3 is trying to be more nimble, but it's a protocol! Ossification is a _feature_ of protocols; for TCP it allows the backbone to evolve without everyone re-implementing everything over and over again at the edges!
The solution is to make TCP like UDP instead, by allowing sockets to not resend missed packets.
Another real solution is to remove the network kernel on linux.