Hacker News Reader

Comment by drewg123 | Hacker News Reader

As a counterpoint to the advantages, the disadvantage is that QUIC is much more expensive on the server side. You loose 25 years of stateless hardware offloads done by the NIC.

Measuring the workload that I care about (Netflix CDN serving static videos), loosing TSO, LRO, and software kTLS, like you would with QUIC, takes a server from serving 90Gb/s @ 62% CPU to serving 35Gb/s and CPU limited. So we'd need essentially 3x as many servers to serve via QUIC as we do to serve via TCP. That has real environmental costs.

And these are old numbers, from servers that don't support hardware kTLS, so the impact of loosing hardware kTLS would make things even worse.

View on HN

(I work for Google, opinions my own)

For YouTube, the CPU cost of QUIC is comparable to TCP, though we did spent years optimizing it. [0] has a nice deep dive.

Other CDN vendors like Fastly seem to have the similar experience [1].

I believe sendmmsg combined with UDP GSO (as discussed, for instance, in [2]) has solved most of the problems that were caused by the absence of features like TSO. From what I understand, most of the benefit of TSO comes not from the hardware acceleration, but rather from the fact that it processes multiple packet as one for most of the transmission code path, meaning that all per-packet operations are only invoked once per chunk (as opposed to once per individual IP packet sent).

[0] https://atscaleconference.com/videos/networking-scale-2019-i...

[1] https://www.fastly.com/blog/measuring-quic-vs-tcp-computatio...

[2] https://blog.cloudflare.com/accelerating-udp-packet-transmis...

GSO is better than nothing, but its still far more expensive than actual hardware TSO that swallows a 64KB send in hardware. At least it was when I did a study of it when I was at Google. This is because it allocates an skb for each MTU sized packet to hold the headers. I think it costs something like 1.5-2.0x real hardware backed TSO, mostly in memory allocation, freeing, and cache misses around header replication.

Also, sendmmsg still touches data, and this has a huge cost. With inline kTLS and sendfile, the CPU never touches data we serve. If nvme drives with big enough controller memory buffers existed, we would not even have to DMA NVME data to host RAM, it could all just be served directly from NVME -> NIC with peer2peer DMA.

Granted, we serve almost entirely static media. I imagine a lot of what YouTube serves is long-tail, and transcoded on demand, and is thus hot in cache. So touching data is not as painful for YouTube as it is for us, since our hot path is already more highly optimized. (eg, our job is easier)

I tried to look at the Networking@Scale link, but I just get a blank page. I wonder if Firefox is blocking something to do with facebook..

> For YouTube

For the CDN/edge this is irrelevant and should not even be part of the discussion. It is obvious it will not change (or will be even better) for them.

The comment you are replying talks exclusively about the "middle boxes". They will have a hardtime with quic, not matter what. (IMHO, a small price to pay, too bad it is something flawed like quic instead of a true distributed solution)

The comment replied to talks about

> Measuring the workload that I care about (Netflix CDN serving static videos)

not middleboxes...

Not a big fan of QUIC (for other reasons) but I didn't consider your very valid points. Not be able to utilize TCP offload engines is another big disadvante of QUIC - but to be fair, I think we will see QUIC offload engines develop over time just as they did with TCP. Maybe those Intel/Altera or AMD/Xilinx deals will pay of for datacenter, for exactly those reasons (quick adaption of emerging network protocols by programming offload engines to FPGAs).

How many of these downsides are inherent to the protocol compared to simply not being optimized yet? I get that a QUIC optimized NIC is unlikely to be right around the corner (these things take time), but is there anything that makes such a thing impossible?

Most applications on the internet are CRUD apps and not everyone is serving Netflix CDN data. TCP hasn't really changed much in the past 25 years due to routers on the internet being strict about its standards. Just look at 'TCP Fast Open', while it was supposed to drastically cut down on connection time in TCP, it can't be used on the internet due to routers behaving erratically. Apple tried it in production and then reverted it.

Since QUIC is based on UDP, it doesn't face the same limitations that TCP does. TCP also suffers from head of line blocking, which QUIC also eliminates. It is also more secure due to header being encrypted, unlike TCP. Plus you also get multiplexing built-in. QUIC is far better than TCP and for serving websites & APIs, it is a much better alternative.

But CRUDs also benefit from load balancing that is going to be much more expensive without hardware support. For the other problems, mostly satisfactory solutions exist.

You can't really solve the head of line blocking problem in TCP

In CRUD apps all head of line problems are caused by abject bloat. Protocol isn't going to fix that.

Interesting. Could this be mitigated at all with QUIC hardware?

It seems like if the tradeoffs are this bad at scale Google wouldn't be pushing QUIC.

It can be mitigated, and companies are working on it. However, there is no NIC offering hardware QUIC protocol offload that I'm currently aware of. Part of what makes it challenging is one of the features of QUIC: That the protocol headers are themselves encrypted.

But do the headers have much to do with power efficiency? Seems like that would mostly hinge on encryption and state machine efficiency.

They have to do with the viability of offloading. You can't give the NIC an encrypted stream and have it handle the protocol stuff if it needs to put things inside the encryption to do so.

The Intel 700 and 800 series NICs have a DDP for QUIC.

Horses for courses. Is anyone suggesting QUIC for gigantic bulk transfers? TCP isn’t great for latency-sensitive small transfers so we will have QUIC. QUIC will not be perfect for everything, like TCP is not perfect for everything.

> Horses for courses. Is anyone suggesting QUIC for gigantic bulk transfers?

Anyone who's looking to replace HTTP/1.1 and HTTP/2 with HTTP/3? AFAIK QUIC is completely integral to that, in order to avoid this issue you'll need a mixed deployment of HTTP/3 and an older version of the standard.

I’m positive that an organization the size of Netflix which has dedicated itself absolutely to pushing 200gbps out of every individual box can figure out how to run HTTP 1.1 forever.

I just hope we don't remove support for HTTP/1.1 from our stacks entirely. 2/3 are great for performance but an order of magnitude more complex.

I imagine that HTTP/2 use will almost entirely move to HTTP/3, and HTTP/1.1 usage will decline but there'll still be plenty of it around.

Cloudflare’s Radar dashboard has recent measurements of HTTP/1, 2, and 3 traffic (near the bottom of the page):

https://radar.cloudflare.com/

Right now, HTTP/1.x is 23% of traffic, HTTP/2 is 75%, and HTTP/3 is just 2%. Since this is about overall traffic (visible to Cloudflare), big sites like Google and Facebook dominate. All that HTTP/3 traffic is probably Chrome users watching YouTube.

"Whee, the magic number went up, better upgrade".

Despite Google meddling with the standards, "HTTP/3" isn't HTTP.

> Despite Google meddling with the standards

Please let that misconception die. I’m not at all fond of Google, but HTTP/3 is not a Google thing: Google is just one (admittedly significant) party involved in making a thing that lots of parties want and have worked together to make. https://news.ycombinator.com/item?id=25286488 is a relevant comment I wrote a couple of months ago.

> ...a thing that lots of parties want and have worked together to make

So,

a) what parties? b) what exactly did they contribute?

Look, yeah, it's a good thing that standards still exist and Google has to play social engineering games instead of suddenly dropping this stuff on us as a fait accompli.

But let's not pretend like Google doesn't own the Internet; it'll save us grief and tears in the future if we see things clearly as they are.

It’s all on public record; take a look in the relevant IETF working group. There are quite a large number of parties that have made meaningful contributions, and I’m too lazy to find and enumerate any. And in line with my linked comment, Google engineers want those contributions, as the end result of such collaboration is consistently significantly superior to what was developed by one party alone. Google is not strong-arming IETF into doing things. There’s widespread (though certainly not universal) consensus that QUIC or an equivalent is a worthy goal, and worth collaborating on to achieve that end.

Look, I wanted something like QUIC before they produced it, because HTTP/2 was better than HTTP/1.1 in most ways, but had the critical TCP HOLB problem that makes it markedly worse than HTTP/1.1 in some situations—and something like QUIC is the only way of fixing that, since the SCTP approach is no longer socially viable. (And before that, I had long wanted something like HTTP/2, because the limitations of HTTP/1.1 were sometimes rather frustrating. —Though I must confess that I wasn’t immediately taken with SPDY and the particular way Google pushed it initially.)

Google doesn’t own the internet. Let it go. Yes, they have an unhealthily large influence on consumer matters—but also nowhere near as large in infrastructural. Google were the ones to advance the initial SPDY and QUIC proposals because their position as massive server operators and manufacturers of a popular browser meant that they were simply the company best-suited to making such initial development (and sure, it served their interests well too).

I'm pretty sure YouTube uses QUIC for video streaming today, which is a similar workload to ours.

Yep: YouTube currently delivers over gQUIC and h3 (IETF QUIC).

We advertise the following alt-svc for most devices:

> h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"

HTTP/3 (over IETF QUIC) is still not as widely deployed due to lesser client support, but that's growing as both Chrome[1], Firefox and Apple (in iOS Safari, when enabled) adopt it and the latest drafts.

[1]: https://blog.chromium.org/2020/10/chrome-is-deploying-http3-...

I think that indicates more about the economics of YouTube than anything. Probably their frontend CPU costs would have to go up by a factor of ten thousand before it even began to approach their costs for compression and storage of user uploads.

> I'm pretty sure YouTube uses QUIC for video streaming today

Is there any media transfer protocol based on QUIC?

Lucas Pardue, who is the IETF QUIC WG chair, has a great talk from Mux's Demuxed conf last year on using QUIC as a transport for UDP streaming: https://www.youtube.com/watch?v=Zdkjd7-EWmQ&feature=emb_titl...

This is not dissimilar from SRT.

I found an article from 4 years ago saying that 20% of youtube's traffic was QUIC: https://www.fiercewireless.com/wireless/youtube-driving-more...

Youtube is delivering video over HTTP. HTTP3, i.e. it is transported over QUIC.

> So we'd need essentially 3x as many servers to serve via QUIC as we do to serve via TCP. That has real environmental costs.

So do ads, and no government seems to care much.

Because other things with environmental cost (which are widely criticized for it!) exist, one shouldn't consider environmental costs?

3x as many servers, it is probably going to be 3x more expensive. Money is something that everyone cares about

Such false equivalence. Everything costs resources to produce. Simply stating that it costs resources, or even just stating how much resources it costs, is completely meaningless on its own.

Pixar movies cost resources, prolonging life with medical treatment costs resources, chocolate candy bars cost resources, operating the Olympic Games costs resources.

In each case, various people believe the resource costs are worth it to get the thing that is produced. Same for bitcoin, same for ads.

You can’t just say they cost resources. If you’re advocating these things shouldn’t exist, you either need to get into the specifics of why they offer value but just not enough to offset their resource costs, or why they don’t offer value, and persuade others to agree with you.

Judging by the success of ads and bitcoin, I think, “these things consume resources” is a pretty weak and useless argument. It would not matter if the resource costs were astronomical or tiny, it’s a specious chain of reasoning in principle.

> You loose 25 years of stateless hardware offloads done by the NIC.

That's a feature.

Otherwise the industry will die off slowly. Case in point, OpenStack first, and later kubernetes, were exactly this kind of innovation. It offers very tiny benefit and a whole slew of plugging points for the industry players to hook upon; but is beneficial enough for mostly everyone to be comfortably endorse a piece of technology that at the time were already starting to decaying... Of course in this case, OpenStack has been a much worse tech because of it was developed by people haven't been the mainstream...

Reply on news.ycombinator.com