How QUIC is displacing TCP for speed | Hacker News Reader

How QUIC is displacing TCP for speed

engineeringatscale.substack.com

149 pointsanimesh371g179 comments

TCP is also concerned with fairness (after Van Jacobson's famous paper[1]). We've long known that you can batter data through the network at speed if you don't care about other users. How does QUIC preserve fairness?

[1] https://inst.eecs.berkeley.edu/~cs162/fa23/static/readings/j...

According to this random document I found on the internet [0], in section 7., QUIC implements a congestion control system similar to TCP New Reno.

[0] https://quicwg.org/base-drafts/rfc9002.html

More precisely - it suggests one. Just like for TCP (see the myriad number of algorithms available for Linux's TCP stack) you can choose whatever congestion control strategy you want.

Fairness? It seems noone gives a shit about it. Most consumers want their data now. I have ETTH 100Mbit (with is very fast to my standards) and I can tell the difference between day, evening and weekend just by pinging my nodes. RTT is stable offhours, but in normal day jitter kicks in. It sad to see RTT jump from 9 to 90ms over major IX. Everything seems to be overbooked horribly.

I know that Internet is best effort network (ATM, anyone remember it?) but Im sure I would preffer slower but more stable internet.

ISP overprovisioning is a completely different issue from fair congestion control algorithms... every device involved can be using totally fair congestion control algorithms and an overprovisioned network will still see degraded performance. I mean that's literally the point of fair congestion control algorithms, make it so that the overprovisioned network still works and everyone's performance degrades roughly evenly!

Slower by how much?

Would 90ms be ok for you?

There you have it!

If you don't want to accidentally get used to 9ms, just add a traffic shaper on your end to slow down your network.

You can't change the fact that during peak activity there are so many people who send so many packets and saturate the bandwidth in such a way that the current fair congestion protocols allocate you the bandwidth and latency profile you observe.

Oh, belive me, I have very decent QoS on my side. Using combination of HTB and sfq+pfifo for traffic. I can download/upload and jitter wont go more than 2ms.

Problem is on ISP side. They seems to have over provisioned network that cannot handle peak loads. Oh well, here we go, everyone selling 300-1000Mbit connections on left and right.

You also seems to confuse bandwith with ping (rtt). Thats not related. I would be happy with 10Mbit connection that have stable RTT.

Latency and bandwidth are not completely unrelated.

When you saturate the available bandwidth you simply cannot send more packets through the same channel so the only option is to queue (or drop and resend later), which means the packet will be delivered later.

You seem to be describing a severe congestion at the ISP side due to your ISP not having enough bandwidth to handle their peak load. Surely, they could work smarter and selectively shape the traffic so that they drop the traffic you think is less important than yours but a) they may have crappy traffic shaping infrastructure b) they may not agree with your characterization of traffic priorities and they will rather not throttle video streaming or whatnot.

EDIT: my point is that it's very hard to actually know what's the cause for the high latency unless you actually work for the ISP. Several causes can have the same observables

That's probably due to the absence of an effective AQM as the bottleneck, rather than the transport protocols being unfair. See https://datatracker.ietf.org/doc/html/rfc7567 for the current best practice

ISPs usually don't perform congestion control by simply dropping packets (and hoping that the affected flows will scale down fairly).

At the edge (where congestion is most likely), they'll usually enforce some type of QoS or just round-robin scheduling, which means whatever greedy congestion control algorithm you use, you'll at most be able to hog 1/N of the available throughput.

If the congestion happens further inside the ISPs backbone, the solution is usually to upgrade that component instead.

Where fairness can matter is inside a home network: If the upstream router doesn’t do anything about it, it is possible for a greedy flow to force out other competing flows. But you don’t need any fancy non-TCP protocols for that: Just open tons of TCP flows. The solution there is to yell at your roommates to stop the torrenting immediately, you have an important Zoom call, damnit!

Congestion control is largely orthogonal to transport protocol design. You can basically slap the general shape of any congestion control algorithm onto whatever transport protocol you want.

Their interaction is largely at the level of: “How easy does the protocol make it to estimate the data channel parameters?”, “What happens in the event of congestion-related failures (packet loss, delay, reorder, etc)?”, and “How do you efficiently recover and adapt to the new channel parameters?”

In all of these regards QUIC is quite a bit better than TCP.

All this QUIC talk makes me think of tape backups. Wonder if a tape backup could be devised which was inspired by network congestion algorithms - i.e. if a section of bad tape was discovered, the same data would get "resent" - to tape! Until it read back correctly (by a read head just behind the write head).

This way, the overall tape data density could be increased even if it would mean occasional dropouts.

Maybe this is already how modern tape drives work?

Transparently remapping good storage over bad storage to deal with faulty sectors is standard practice in hard disk drives and solid state drives. It was historically done in software, though these days it is largely done in the drive microcontrollers directly. Tape drives almost certainly use similar techniques.

Wikipedia confirms that IBM introduced this in 1958.

> Separate read/write heads providing transparent read-after-write verification.

There's a general IBM theme of "whatever tech we come up with", IBM did it first, a very long time ago, they have their own name for it, and everyone else "discovers" it years or decades later.

> It was historically done in software, though these days it is largely done in the drive microcontrollers directly.

It's still software on the drive microcontrollers, though.

The only difference is it's software you can't see, reason about, upgrade, improve, or fix.

If a modern "one-shot" tape drive encounters an error during write, it will write the data again as the tape is moving. You lose a little bit of total capacity, but the data will still get written. This is why you always plan to write a little less than the full capacity of your tape.

Newer LTOs may behave a little differently, I haven't kept up with newer stuff here.

Buuuut.... If you're running a big web-app (eg. tiktok) you have a business incentive to not be fair.

If your servers can push more data down the users slow internet connection, your service will get better load times, while competitors services sharing the same network link (ie. your brother in the next door bedroom on youtube) will stutter and lose users.

You can already do that with TCP so it's not that big of a change.

I'm not sure what you mean. If you're talking about video delivery, TikTok is using HLS, so the client is initiating the download. Are you saying that the server will attempt to multiplex across QUIC streams and overwhelm the client?

The server can deliberately send packets down those QUIC streams faster than the protocol would normally.

Typically that results in the client getting their download faster, but harming other users on the network (whose downloads slow down)

That doesn't make any sense to me. What do you mean by "on the network"? Is your claim that using UDP will somehow saturate a shared circuit between multiple subscribers? What's stopping the switch from just dropping excess packets for one destination?

> your claim that using UDP will somehow saturate a shared circuit between multiple subscribers?

Yes. And it isn't specific to UDP. Basically any file transfer will keep going faster till at least one network segment is saturated. At that point, if the server keeps sending data a little faster than a regular congestion control algorithm would allow, you will cause packets from other users streams to be dropped.

Okay so that's the same thing you can do with TCP or any other transport atop IP, so not sure what QUIC is doing special here. Congestion control is handled at the transport layer.

I think the part not said here is that ISP routers have ways to throttle per-user throughput, but home routers do not. QUIC has its own congestion control, but if it didn't, that'd be a problem.

Thanks that makes more sense.

It seems like a bug in the middlebox if it requires all end points to cooperate and doesn't throttle anyone.

I thought the point of congestion control was to slow down pointless sending when an overloaded middle box is dropping our packets and saying "buddy take the hint", not to actually yield to other traffic? The network tells us how much capacity we have, trying to push more would just increase our own stream's packet loss. ... Right?

I don't think this article was very good and seems to be written by someone that doesn't really know what they're talking about, or by someone that does but "dumbed down" to a degree where it's conclusion inaccurate.

I think HTTP3 is probably doomed for a IPv6 like existence for a long while. While everyone claims that TCP is apparently "too slow", the vast majority of corporate/enterprise settings will just block it.

It seems like a technology built by the big players who want to shave a few cycles off each connection and save $millions rather than a practical standard.

Do I want to use HTTP3 at home? Yeah, sounds cool.

Will I be able to use it at work? Probably not for 5+ years.

HTTP3 doesn't have the same chicken-or-egg problem as IPv6. It runs over UDP. IPv6 is a full protocol rev which takes a lot longer to accomplish. Once a large multi-vendor network is deployed revving the base protocol is really hard.

Not to mention the companies backing it are so large and critical they can simply mandate it and enterprises will just have to deal with it, somehow.

so far, there is no reason to mandate http/3. There is barely any infrastructure cost to running a dual HTTP/1.1 and HTTP/3 setup.

And the two are so similar to application developers that there is no reason to mandate anyone move over.

However... It only takes one killer feature not compatible with HTTP/1.1 (eg. realtime VR with a stream per eye, and it turns out HTTP/1.1 can't do realtime bidirectional streams), and suddenly everyone will start turning off HTTP/1.1 services and replacing it with a page saying "please upgrade your web browser".

That part is great. It uses UDP, which means it helps prevent UDP from being sidelined or blocked by middleboxes. That in turn helps prevent the Internet from becoming a de-facto HTTP-only network, which was a trend that was kind of happening in the 20-teens and had me concerned.

I agree, I just hope we don't see the opposite problem happen where we end up in a world where HTTP/1.1 over TCP goes away. I implemented a HTTP/1.1 server with range request support in JavaScript in a few hours last week. It's incomplete and likely rather insecure, but it works with curl and browsers. There's something important about having this ability.

HTTP/1.1 going away is less likely than IPv4 going away.

Yeah but

> Palo Alto Networks recommends creating a security policy in the firewall to block the QUIC application. With the QUIC traffic getting blocked by the Firewall, the Chrome browser will fall back to using traditional TLS/SSL. Note that this will not cause the user to lose any functionality on their browser. Firewall gains better visibility and control of Google applications with or without the SSL decryption enabled.

https://docs.paloaltonetworks.com/best-practices/10-2/decryp...

https://knowledgebase.paloaltonetworks.com/KCSArticleDetail?...

Fortinet also recommends blocking it.

> However, due to the protocol is still in an experimental stage, it is not supported by Fortinet and causes some issues when SSL inspection profiles are needed to block specific websites or applications provided by Google itself.

https://community.fortinet.com/t5/FortiGate/Technical-Note-D...

And given how trivial and likely udp/443 is already blocked because the only reason prior to QUIC you would be hitting udp/443 is some malware trying to get past the firewall and hoping it's open without inspection turned on you're fighting an uphill battle.

Is this the same Fortinet that just did a stock pump by releasing fake news stories about toothbrush DDoS attacks?:

https://www.forbes.com/sites/daveywinder/2024/02/08/surprisi...

I wonder how valid their claims are for anything, if they just make up things for press and publicity.

Do I need a Fortinet firewall for my tooth brush?

> IPv6 is a full protocol rev which takes a lot longer to accomplish

And maybe will never happen. There are alternatives for a lot of what you get from IPv6, such as NAT (terrible but works for most people's needs) and SNI routing (I'm more bullish on this).

Are there any examples of protocols that took a similarly long time to achieve adoption but eventually did saturate the market? That would be hopeful.

> Are there any examples of protocols that took a similarly long time to achieve adoption but eventually did saturate the market? That would be hopeful.

There are quite a few, but people probably haven't heard of them, because either they were involved below the Internet (think: physical links) or were used for things like telephony or other interconnects. There are also other interfaces that involve communication but aren't Internet or telephone like (think things like ic2 or canbus, etc).

If it's still not adopted more in 5-10 years, I wonder if they'll make what everyone wanted all along: IPv4 with longer addresses.

That would be absurd. Why introduce a completely new protocol with 0 hardware or software support? Who would possibly adopt this over IPv6?

It would be possible to do basically this with the existing IPv6 header and much of the v6 infra. All existing IPv4 hosts get the same /32 on v6, and any NATs stay intact. On the UI side, keep the ipv4 address format like 8.8.8.8. Going to V6 is very low-risk, low-impact this way. V4 packets can even be converted to V6 and vice versa because it's the same address space at first. Once there's sufficient adoption of V6, individual network operators can break things up into /40, /48, etc if needed, which wouldn't be reachable via V4 anymore*. Back to UI, a /40 would look like 8.8.8.8.4.

Unlike today's v6 which is a totally separate network with new routes and randomized addresses. It also has v6-mapped-v4 addresses, but that's different.

* If you really want some more temporary compatibility, /40 would still be sorta reachable from V4 if you added a special kind of NAT that encodes the extra 8 address bits into the port number, but idk if that'd be palatable.

NAT isn’t an alternative. It breaks end to end connectivity and doesn’t scale well.

The default and recommended way of using IPv6 is to have a default-deny-inbound firewall. That also breaks end to end connectivity.

Even if HTTP/3 had gone the route of registering QUIC as a new transport protocol instead of UDP, I think it would have still been at least one or two orders of magnitude easier to deploy than IPv6 was (though still much harder than going over UDP is, I'm not suggesting this was a bad choice in any way!).

They would have had problems with kernel updates and with devices that make assumptions and more paranoid security policies, but they would have probably been able to convince most people to allow their packets to flow after a few years. Changing IPv6 is a much more fundamental change to the Internet as a whole.

I think assigning a new IP protocol number would have made QUIC a complete non-starter, and for almost no benefit compared to just encapsulating it in UDP. That's why WebRTC encapsulates SCTP (for data connections) in UDP, why ESP (for IPsec) is encapsulated in UDP etc.

> they would have probably been able to convince most people to allow their packets to flow after a few years

Looking at all these stateful NATs in network cabinets that nobody has looked at for years, I highly, highly doubt it.

I'm annoyed that the comment above yours dismisses the entire article based on the false assumption that QUIC is its own L4 protocol. I was fooled for a second, then I actually read the article, where it clearly says QUIC runs over UDP. So there's no chicken-and-egg.

A lot of firewalls are blocking UDP and unknown L4 protocols so QUIC is blocked either way. QUIC can only work because happy eyeballs falls back to HTTP/2.

UDP is definitely better for NATs since they don't know how to translate unknown protocols (e.g. SCTP).

Which firewalls are broadly blocking UDP? I haven't seen that on a residential router or ISP. Lots of things rely on UDP already.

Corporate firewalls tend to be much more restrictive than consumer.

Yeah, if it's just corporate falling back to HTTP2 (or maybe more realistically, HTTP1), then I think it's alright. Same corporate users delayed leaving unencrypted HTTP, broken versions of TLS, Windows XP, etc yet everyone else adopted anyway. They'll either follow suit eventually or work around it.

If you use Chromium or Firefox, you may have used HTTP/3 and not realized. It happened to me!

https://ifconfig.net/ uses HTTP/3 by virtue of Cloudflare's proxy using it. I'm on Firefox 115, half a year old, and the network inspector says I connect to ifconfig over HTTP/3.

The rollout's been really smooth and quiet. Browsers do the same "happy eyeballs" optimization as they did for HTTP/2 and SPDY, racing QUIC with TCP and using which-ever one connects first, or maybe dropping TCP if QUIC connects a couple milliseconds later.

Hell, CF actually rolled out HTTP/3 before the pandemic, in 2019 https://blog.cloudflare.com/http3-the-past-present-and-futur...

HTTP3 works fine. It's used by Cloudflare, which (unfortunately) means that half the websites you visit probably use it. It's available in nginx, caddy, and a bunch of other servers.

Sure, most servers will just support the protocol at the reverse proxy level and lack the optimisations that make HTTP3 faster for the big cloud providers that stand to gain most, but you can just set up HTTP3 on your own server if you want.

I don't know what you're using at work that prevents you from using HTTP3. I'm guessing you're referring to one of those awful middleboxes or some kind of firewall that blocks outgoing UDP traffic. Luckily, that stuff isn't relevant for most connections on the internet.

If you put an HTTP3 nginx server in front of HTTP1 applications you get a lot of the UX benefits… parallel requests that make bundling, and to an extent tools like GraphQL and Falcon, somewhat obsolete and in turn applications much less complicated.

Yes, it saves server resources at extreme scales. The client-side savings are purely theoretical. If their JS bloatware crap takes 2s to respond to a click, the 3x 30ms network roundtrip to fetch a resource with a new HTTPS connection hardly matters.

ipv6 is very, very widely deployed in mobile networks. Download some net tools and check out your phone's interface table.

Also v6 did a bunch of things right. NDP and link local addresses for v6 actually work. The fragmentation changes were 100% the right thing to do. v6 extensions like SR and uSID are both solid and consolidate a lot of stupid stuff into something clean.

v6 had a bad, rocky start. A lot of that was completely inadequate attention to the transition, and the vendors fucked up almost everything for years, on their own gear and interop, but at this point v6 is fine.

Would I launch a v6 only service? No, because at this point the SPs are the problem. Verizon as of a few years ago and maybe now still didn't do v6 to any of their home users. So many things like that.

> Verizon as of a few years ago and maybe now still didn't do v6 to any of their home users. So many things like that.

My FIOS home connection has had IPv6 since some time last year. Consumer ISPs are actually quite motivated to offer IPv6 since it gives them a path forward without an IPv4 address per customer better than CG-NAT. Many consumers also just use their ISP's CPE, which makes this shift easier as well.

I think the bigger obstacle will be managed corporate connections (for Wi-Fi or other client access connectivity) that are only upgraded once every few years and just don't support IPv6.

I finally got IPv6 working on my home network this week. /60 prefix from the ISP, Slaac with PD for addresses, RA for DNS, unbound DNS working just fine.

OPNsense works great! This is something I've wanted to get working for years.

Ipv6 isn't just an upgraded protocol over ipv4. It could've been, but it's not. It's a totally separate network with its own routing. So just changing all hardware along the way to support ipv6 doesn't mean it's adopted.

IPv6 isn’t just an upgraded protocol because the designers knew they didn’t want to go through the painful process again and wanted to get it right.

The write that the tcp three way handshake is syn-ack-synack.

I don't trust it.

HTTP/3 may have good speed for webpages that trigger many HTTP requests to source ads and support tracking/telemetry from multiple, disparate servers controlled by different entitites. (Makes sense since HTTP/3 was designed by an advertising company.) But HTTP/3 speed is not any better than HTTP/1.1 speed for downloading a file or a series of files from the same server in a single TCP connection. Some have said it's worse. (HTTP/1.1 was not designed by an advertising company so if it sucks for ads/tracking/telemetry then that makes perfect sense.)

https://news.ycombinator.com/item?id=39164152

I don't think this article is very well-written. It gets confused about whether QUIC has a handshake or not (it does). And it conflates zero round-trip time with combining the TCP/TLS handshakes together.

QUIC doesn't have a handshake which is another reason for it being fast

The article says this repeatedly, but it's also wrong about that. QUIC doesn't use exactly the same three-way handshake at the beginning of the session -- but it uses a handshake that lasts at least 3 packets.

https://quic.xargs.org/

I came to mention the same. Diagrams have redundant information, examples are badly picked, there's sentences with little to no value... I don't know if it's lack of care, the author not writing in his native language, or excess of GPT.

> QUIC works on top of UDP. It overcomes the limitations of TCP by using UDP. It’s just a layer or wrapper on top of UDP.

Makes me want to stop reading.

It's a bit perplexing that an article that make some claims about QUIC speed over TCP has exactly zero benchmarks, numbers, anything to back that up besides theory.

I could be inclined to believe it but I'd like to know by which factor, in which circumstances, with real examples and numbers.

What would you want to compare? Google Quiche vs. Linux? Chrome vs. Windows TCP? What version of Linux, Windows, or Chrome? Under how much delay, loss, etc? There's a large parameter space to sweep.

Uh, how about "something" vs. "nothing".

Start with QUIC vs. TCP. Same OS, same client/server software.

Edit: since the system says "I'm posting too quickly, slow down" despite not posting for an hour because they don't use accurate error messages here, let me say :

I find it ironic that so many keystrokes below are used in a meta-discussion about the quality of comments here, when others are posed a legitimate question about the topic. Nevermind the article they seem to defend gets the order of a TCP 3 way handshake wrong.

And since I can't respond to them, I can only d*wnv*te or fl*g them.

Real-world test. Load Facebook on an iPhone in HTTP3 vs 2 or 1. Load YouTube in Chrome on Windows. At least tell me if the page ends up being rendered more quickly. This won't capture things like server load, though.

We already have public first-party claims for both YouTube and Google Search performance using QUIC on mobile and fixed endpoints. I don't think anyone disputes these.

https://youtu.be/BazWPeUGS8M?t=1949

QUIC: let's build an application-specific protocol on top of UDP and call it a day.

So really, is it faster? Does it reduce the amount of load on network devices? Does it allow a server to serve more connections more quickly than the equivalent HTTP/2.0 stack? Does it make my web app faster?

I mean, for 99% of Wordpress sites the problem is Wordpress, not the transport protocol. For a lot of the web the problem is client-side rendering issues.

QUIC may solve a problem, but is it a problem in real life or a thought experiment that got folded into a standard?

Most of the web traffic is BigTech and big 5 and they see significant cost and energy reductions using QUIC. Nobody cares about Wordpress sites, as it does not make a dent.

And most of the flows on this planet are inside hyperscaler datacenters. QUIC demonstrably solves problems that TCP causes inside datacenters.

If your Wordpress site is hosted by one of the big hosting providers and/or behind Cloudflare, it's quite likely to already use QUIC.

> Does it reduce the amount of load on network devices?

This is not a goal of QUIC: Ideally, network devices between two TCP hosts don't do anything to TCP segments that they wouldn't also do to UDP datagrams, so there shouldn't be much difference.

> I mean, for 99% of Wordpress sites the problem is Wordpress, not the transport protocol.

Addressing Wordpress problems or rendering issues is also not really a stated goal of QUIC.

It does solve some real TCP inefficiencies though, especially when used on older OS/kernel versions that'll probably never be updated to make use of some of the newer developments there (Bufferbloat avoidance is a big one, for example).

These might not be very high on your list, but they absolutely are on someone's. And as long as you don't host your own webserver, you don't need to do anything to benefit from the results, since your hoster/CDN/proxy will just transparently provide them for you.

I don't have benchmarks but I built a protocol that QUIC later turned out to be eerily similar to. It was a direct replacement for TCP and we used it to provide an additional layer of encryption for all traffic on a given network between two points.

Latency was exceptionally improved. Web pages felt like the loaded faster and at the very least, users could not tell that they were using an encrypted connection.

The protocol essentially worked using a fast-ACK protocol that would preemptively request retransmits (and was occasionally wrong). This enabled it to use connectionless UDP protocol as the underlying transport mechanism. There is, of course, a cost for reduced latency. That cost was slightly higher bandwidth utilization on the network. This was suboptimal for long-lived streams (media and other downloads) so we tried to fault over to ordinary TCP in these instances.

From the diagram they draw QUIC is supposed to solve transport layer problems (reliability, congestion, security). Perhaps some session layer issues slip in as well, but it looks like you don't have to put HTTP on top of it.

I have no clue why it's built on UDP though, do we just accept that the time has passed for truly new network protocols?

It's built on UDP because UDP is simple enough to basically be perfect. It's header is 8 bytes, of which the first 6 are obviously necessary and the last 2 are pretty nice. If you were re-doing the protocol, how would you change it to be more efficient?

You say they are obvious but it seems to me that the source port and destination port of UDP are going to be fixed for QUIC, so I'm not too sure those are necessary. Especially since QUIC has connection IDs instead. I don't know QUIC on a deep enough level to figure out if the length and checksum are still needed.

I mean I get why you'd tunnel it through a UDP connection, since replacing every single router on the planet no longer seems feasible, but if you want to call it a transport level protocol that's kind of what you'd have to do.

> I have no clue why it's built on UDP though, do we just accept that the time has passed for truly new network protocols?

Yes. According to my research, the Internets middleboxes are ossified and if they see a packet that’s not UDP or TCP they will silently drop it. SCTP was an past attempt to introduce a new protocol AFAIK https://en.wikipedia.org/wiki/Stream_Control_Transmission_Pr...

Combining the TLS and TCP handshakes into a single request amounts to probably less than 50ms latency. For a major web hosting provider, this might add up. For most just cases, it's trivial.

50ms in the good case, but for people halfway across the world or on a slow connection (e.g. geostationary satellite or bad 4g) each round trip can pretty easily be ~100-500ms. It's not about reducing server load, but improving user experience.

True. But, the application data is still going to be many more packets than the few you save during TCP/TLS setup. I imagine the user in that scenario would not see a significant difference, but I'd be curious to know for sure. Retransmits being smaller/less frequent is an improvement, but I'm not sure how much - latency due to geographic distance is pretty much built in. I just might google if there's any data on this. Since major providers are adoping it, I have to assume there is an improvement.

I feel the main limitation over here is hardware optimisation support.

With TCP you have the congestion control algo baked in hardware, tcp segment and checksum offload. You can pass things directly to the NIC for massive latency, bandwidth and offloading processing away from the cpu.

A properly tuned system with a user space networking stack with ctpio, hardware offload and proper tuning beats the pants of QUIC for latency.

It is possible to get some of the same benefits I guess with GSO. In any case, the slow hardware support here I suspect is a bottleneck. You may not get much benefit given more layers are binary/encrypted and not visible to hardware.

The above is not relevant for hyperscalers like Google I imagine where they can make the hardware and the sheer amount of customers offer bandwidth benefits.

The internet itself is not properly tuned and distance is an issue. On a local lan, sure a TCP tuned system will do amazing. On the public internet where who knows where your packet is going it's an optimal but not tuned system with latency TCP wasn't designed for. Also a modern CPU with SIMD/AVX can handle a lot of traffic in user space. UDP also has a checksum in it's packet header.

For hyperscalers in the cloud and low latency finance, tcp is likely still king.

You do not want things running on CPU as that is compute that could be sold to someone else.

This needs an easy way to offload to a specialised hardware accelerator.

https://aws.amazon.com/blogs/aws/new-http-3-support-for-amaz...

Seems AWS is doing just fine with it. They also have built their own custom chipsets though I don't see a mention of it offloading QUIC to those chips.

AWS in some ways wasn't really ready for something like QUIC to emerge. Nitro networking does, or did, penalize all UDP flows. Which obviously is not great for QUIC.

Hyperscalers in the cloud and low latency finance will drive the innovation to make HW offload a thing sooner. For the rest of us normal folk, we can likely live with CPU managed packets until that happens, and then take advantage of the optimization benefits when they make it to us.

I think it's fair to call this out, but my initial read, and the reason why I'm responding, is because it seems like you're saying that it's non-viable.

Uh, the fact is that the internet works and has been working pretty well for the last few decades.

While you may have issues with certain aspects of TCP and its behavior over WAN, your issues have no practical significance in real life.

> pretty well

Doing a lot of heavy lifting right there. My experience (across many countries, travels, etc) is that internet (and things in general) just barely work to a barely tolerable level. Not a day where a coworker doesn't get some internet issue, some sudden disconnection, some weird TURN route killing latency..

I believe you but as a networking noob, could someone tell me how segmentation and old checksum algorithms are significant compared to TLS overhead?

Is it because TLS is hardware accelerated with AES instructions or something already?

TLS offload also exists and is normally implemented on NIC too.

Basic goal here is not to process this on CPU as that is slow and compute that could be used for user apps/customers.

In my experience, congestion control in hardware would be the very last thing I would want. Everything needs to be pushed as far toward the edges of the system as possible. This is what quic offers.

For what? It has been done for a while now.

The fastest, lowest latency mechanisms will always offload to accelerator cards in hardware.

The NIC does not have the information required to make machine-wide optimal decisions about congestion control. It doesn't know what flows are critical at the application level. NICs also can't know whether you want to optimize median or tail latency for a given flow. Only the application knows and that's where the congestion control algorithm needs to run.

A lot of NICs are essentially just as software modems, with the CPU handling this anyhow. Even server hardware sometimes has these lame NIC chips on them.

Be careful to rid the specs of the hardware you buy, otherwise it will indeed be your CPU doing all that work.

With TCP you have the congestion control algo baked in hardware

This part isn't correct, and you wouldn't want e.g. NewReno baked in to your NIC and preventing you from using CUBIC or BBR. It's true that TCP benefits more from NIC offloads than QUIC but most places (besides Netflix) aren't driving enough WAN traffic per server to matter.

The chart at the bottom: So google switched it on, and then, let us guess, Facebook and Netflix. Afterwards no growth for a year. Displacing looks different to me. HTTP/2 is growing at cost of HTTP/1 ... so the only conclusion I can draw here, is that HTTP/3 adoption has halted. Do not read that negatively. The users who really needed it (like Google, Facebook, Netfix, ...) are using it and the rest, has it very low on the priority list if at all.

I have my doubts that everyone needs HTTP/3, with UDP traffic has network device wise also its disadvantages and library availability and complexity should be lower for HTTP/1 and /2 for the foreseeable future.

Not disagreeing, but adding: Cloudflare also uses it between browser and proxy, which is a huge amount non-trivial amount of the small internet.

I'm a fan of QUIC, but these articles are always heavy on explanations and light on data. I understand in theory how head-of-line blocking can cause serious issues on a lossy network, but by this point I would expect to see a ton of data backing that up in real-world usage from Google and Cloudflare.

One specific question I've had is on a lossy network are there really that many situations where you would have packet loss on one QUIC stream but not most/all the others? I don't doubt that's true but I would love to see a breakdown.

Also, what's the crossover point between opening multiple TCP streams and doing round robin across them? Maybe you only need 3-4 TCP connections to estimate the HOL advantages of QUIC.

I will admit fewer RTT handshakes is a more obvious win.

> One specific question I've had is on a lossy network are there really that many situations where you would have packet loss on one QUIC stream but not most/all the others? I don't doubt that's true but I would love to see a breakdown.

If an intermediate router is using Randomized Early Detection (RED) and is congested within the randomized packed drop region, then you could easily see one random packet in the middle of a stream dropped, with other streams unaffected.

> One specific question I've had is on a lossy network are there really that many situations where you would have packet loss on one QUIC stream but not most/all the others?

This isn't the advantage. The advantage is that if you have 10 different streams and a low loss rate, the impact of each packet lost would be much smaller. Consider a voip call with an architecture where each person's audio is served as a separate stream from a central server. With head of line blocking, every packet lossed causes a slight stutter in the whole conversation because even if the lost packet is from someone not talking, you have to wait for that packet of silence before decoding the packet from the person speaking. With QUIC, the extra latency only affects the individual stream, so the conversation won't stutter when the person not talking wasn't the one who's packet got lost.

What I'm questioning is how likely it is to lose a single packet on one stream. I understand why in theory that would be bad and QUIC would help, but I'm curious how often it actually happens. Sibling provided at least one example but I would be interested to see some data.

QUIC is very exciting, after seeing what it did for latency in Cloudflare network and Cloudflare workers, I can't wait to finally see it in Deno 1.41[0].

[0] https://github.com/denoland/deno/pull/21942#issuecomment-192...

I’m a bit concerned that the ease of modifying the parameters will lead to Congestion Collapse.

It’s been possible to cheat with TCP but most people don’t bother because it would require a kernel recompilation.

Or did this ship sail a while ago with various other UDP-based reliable transports, streaming media, etc?

If congestion collapse happens on the wider internet, I predict that it would take mere hours for sysadmins to come up with a temporary fix, and within months it'll become common for all core routers to have bloom filters of common src/dst IP pairs and de-prioritize anyone sending more than their fair share of traffic on a congested link.

99% of QUIC traffic is coming from a handful of CDNs (Netflix, Google, Meta, etc.) and they have no incentive to collapse the Internet.

TCP also supports allowing people you've never met to connect to your IP without you getting the permission of a third party first and regularly. QUIC requires continued permission being allowed by some third party corporation. Baking the requirement for CA TLS into the protocol is fine for corporate uses but the internet is far more than just corporate uses.

Sure QUIC may be faster, but only as long as you keep getting permission from your TLS CA. Once that stops it's very, very, slow. QUIC is fragile and like other CA TLS things: anything built with it will have a lifetime of only a few years without being updated.

Do we really want to base our web protocol (and other things) on a system that will only allow machines updated in the last $fewyears to participate?

Additionally, the claims of the use rate for the various HTTP versions is flawed in that it is not going per webserver or domain, but instead just counting traffic and of course the megacorp flows it knows about will dominate (they're not checking every webserver). Looking where the light is fallacy.

> Sure QUIC may be faster, but only as long as you keep getting permission from your TLS CA. Once that stops it's very, very, slow. QUIC is fragile and like other CA TLS things: anything built with it will have a lifetime of only a few years without being updated.

For me, letsencrypt has been set-and-forget for at least 5 years. I use it with nginx, which is in front of several servers including a Jetty 5 server. Jetty 6 was released in 2006. You can configure which CAs you trust as well, so TLS isn't really any more of a centralization problem than DNS is.

What did you do when acme1 was deprecated (for acme2) and stopped working with LetsEncrypt? That certainly required changes in your setup that would break otherwise. How did you handle the expiration of the LE root cert? Maybe can still access your TLS wrapped site but people using an RSS reader who's cert store was set up in 2019 can't. All these things, both on your side and the client side apply very, very broadly. It may work for you from your setup but I can assure you that isn't universal or even normal.

I love LE and I think it's the least worst solution. But mandating CA TLS into a protocol is very bad for human use cases. HTTP+HTTPS prevents this fragility. HTTP/3 with it baked in is fragile.

Also, you don't need DNS to communicate with an IP.

I didn't even know about the acme1 deprecation until you just now told me. Apparently certbot 0.31.0 handled it fine.

You're right that DNS isn't architecturally analogous to TLS. The way it's analogous is that if the central infrastructure goes away, then people have to do something more technical than usual to get things working again. Yes, using an IP address string in place of a hostname is simpler than setting up an alternate CA, but broadly it's the same idea, a centralized dependency with an inconvenient workaround.

>certbot

I'm not surprised the flagship certbot automagically handled the deprecation since you apparently consistently update it. But many setups, especially those without auto-updates and other moving parts, just stopped working. And certbot would've too if you hadn't have made sure it was up to date. And some acme clients never supported 2 and just died. It will happen again.

>a centralized dependency with an inconvenient workaround.

Except there's no workaround with QUIC based HTTP/3 because all the libs people use come without the flags to allow self signed certs. So 99% of browsers/HTTP using software released is stuck with CA TLS or no communication at all.

It doesn't matter if you set up a custom CA for internal use or if you enable it in your custom http/3 lib build of your custom build of $browser. People still won't be able to visit your hosted HTTP/3 site with your custom CA. Not without you installing your root cert in the browser or system cert store of most people on Earth. That seems unlikely to happen.

Is it just me or the graph at the end of the article is not really showing rising adoption in general. It basically shows a couple of spikes - each corresponding to some big company switching their edge servers to HTTP/3 like Alphabet, Meta, Cloudflare. There is no gradual increase.

Furthermore, I don't see easy solutions yet for some problems with QUIC - for example browsers still try to establish a TCP connection first unless they know for sure the server supports QUIC. Proxy support for HTTP/3 is still in its infancy, but for many corporate envuronments it is a hard requirement.

So outside of the biggest websites, which I admit also take up a large chunk of the network traffic, is QUIC really replacing TCP in the general Internet?

The adoption would be gradual and few years down the line, it would be QUIC. The same was true for HTTP/1 and HTTP/2.

Yep, and the graph looks like the HTTP 3 adoption spikes are taking from HTTP 2. It's basically one Google protocol replacing the previous one.

And the original and simpler "real" HTTP 1.1 is still going strong.

Anyway, the quicker we kill HTTP 2, the best. So, IMO, this is one of the best possible scenarios.

> for example browsers still try to establish a TCP connection first unless they know for sure the server supports QUIC

As you alluded to here, you can hint a networking implementation with QUIC supporting servers. This feels similar in practice to HSTS Preloading, most of the benefit of full HSTS comes from a small number O(100k? 1m?) of domains being pre-loaded as HSTS supporting, and that's just distributed with browsers now, a fairly straightforward solution. The same could probably be applied for QUIC.

> which I admit also take up a large chunk of the network traffic, is QUIC really replacing TCP in the general Internet?

I guess this depends on whether you're looking at aggregated traffic, or distinct traffic destinations. Neither of those is more important right! If YouTube/Netflix move to QUIC, that's a very significant amount of benefit for the internet and users. Equally if all wordpress sites on shared hosting disappeared because TCP was no longer supported, that would also be a very significant impact. I think the headline saying "QUIC is displacing TCP for speed" is very fair, but over-extrapolating from this would be going too far.

Curl HTTP/3 performance (https://news.ycombinator.com/item?id=39163948) shows HTTP/3 is the slowest performer compared to HTTP2 and HTTP1.1

This paper from 2017 showed QUIC performance was poor: https://arxiv.org/pdf/2310.09423.pdf

With no evidence to the opposite, the article's claim that QUIC is displacing TCP for speed is dubious at best.

The comments of that article cite a couple things:

* The version of Caddy used for the test was missing several recent updates that significantly improve its performance. * Performance improvements for QUIC have always been about removing/recovering from dynamic network connections, and are about running connections at scale. Simple HTTP 1 requests in environments where there are far less shenanigans going on will always be fast. (lib)curl powered requests being single threaded requesting across the local network stack are not representative of the types of performance gains to expect in an HTTP/3 world.

If localhost is your use case, these results are for you. Many others may be operating on real networks.

Perfect, lossless, zero-delay networks are of no interest when evaluating protocols. Under those conditions literally any protocol can do the job. The exercise only becomes interesting in the real world where delay and loss exist.

It's weird that there's still no QUIC or HTTP3 in Node.js

Best opportunity to contribute to a new NPM package for QUIC/HTTP3

Node.js shouldn't be used in the first place, just like any server-side JS. But more importantly, put your hacky language-specific servers behind a webserver like nginx.

Not really. HTTP3 has little benefit (if any) when talking to a backend server, plus browsers only support QUIC with TLS and you probably don't want to waste CPU cycles terminating TLS on an application server and leave it load balancer (that isn't written in nodejs and supports HTTP3)

"The big a-holes implemented it so you should use it too" ?

That's what I get from the article.

What are the advantages of that thing if you're not at Google/Facebook scale?

What happens when you tcpdump QUIC? Hmmm..

edit: "tcpdump -t quic", and I also see that it's supported (as expected) by Wireshark.

I got bit hard doing network troubleshooting with Wireshark. I was trying to debug connectivity issues and was filtering for HTTP packets, but that was excluding QUIC! Turns out my packets were QUICked through Cloudflare. Figuring that out when hair is on fire isn't my preferred time.

It is currently almost certainly not faster in all cases.

https://github.com/icing/blog/blob/main/curl-h3-performance....

You either use TCP or re-implement TCP over UDP i.e. QUIC

Yes, that is what QUIC has done.

Yep. And you need that, becuase your website, that cotains maybe 1kB of useful info (eg. news article), now needs to transfer tens of megabytes of random garbage, needs you to connect to 50 different external services, and where a simple even a basic share button, that could be implemented with an 'a href' pointing to "http: //twitter.com/share/?url=http....", now needs 3rd party cookies and external resources to track your users.

So in a real-life example, this is like a movie-ticket seller optimizing the ball bearings in a shopping cart that you need to buy those tickets, because they want you to carry a bunch of flyers and gps trackers with you, while all you need is a small paper ticket.

This response is so on-point I might have to frame and hang it on the wall.

Then in a few generations when the world comes back to its senses, some historian will find it and laugh at how dumb and broken our generation was.

The whole point of QUIC is to re-implement TCP to be better than TCP.

most tcp work is done in the network hardware these days isnt it?

how can a userpace protocol possibly hope to be faster than that?

"Fast" here means high throughput and/or low latency, not low on compute resources.

If you're compute-bound (or using a userspace transport layer would make you compute-bound), this might not be a great choice, but many network-facing services aren't.

No, TCP is not in hardware and Linux (the most popular server OS) has a policy to never allow full TCP offload.

Not convinced, but sounds like you know what you are talking about

nginx has quic support now https://nginx.org/en/docs/quic.html

Do you fancy knocking up some benchmarks of typical server loads on accross some 100mbs/1gbs/10gbs connections and chrome headless?

As I said in another comment, I still expect TCP to use fewer cycles than QUIC due to kernel optimizations and partial NIC offload. IIRC people have done benchmarks showing this. Saturating 10G or even 100G with either protocol should be easy enough these days but if you want to do 400G or more from a single server then it matters.

Strong article so far, I'm still reading, but I found it rather jarring to have a cartoon in the middle with a white couple cherishing a baby and a black woman killing one...

"killing"

It's a weird an uninformative illustration.

If this were a photo, would you expect the infant launched high in the air like a basketball to survive?

Do you expect all memes and jokes to be absolutely harmless and 100% politically correct for every nation/culture/race/gender/<group>?

When doing technical or business work - YES

If for entertainment then things can be unsettling or tuned to one group.

I really do not want to work with the person(s) who downvoted this. These sort of 'jokes' are part of what makes a hostile work environment.

Luckily, this is an internet forum and you don't actually need work with that person.

> I really do not want to work with the person(s) who downvoted this. These sort of 'jokes' are part of what makes a hostile work environment.

So does oversensitivity and overreactions.

You might ask a lot of people in minorities etc if this is an overreaction to "banter".

I have not been on the end of this being a white male.

I expect memes not to be blatantly racist, particularly when they are attached to a technical article.

It's only racist if you are a racist. You brought up skin color where it doesn't matter. Work on yourself.

Maybe you’re the only one that saw race in this picture

I'm confused. Is it your claim that the brown woman throwing the brown baby like a basketball is white?

What he is saying is he didn't pay attention to that detail, for one reason or another.

It seems rather likely it was adapted from what was originally a racist meme format, but I doubt the author is aware of that or made a conscious decision to depict it like that. Memes often get adapted with so many edits that their original meaning vanishes. I'm glad we can plagiarize Stonetoss to make positive edits.

Why is there always someone who needs to read things in like this where nothing exists?

It's somehow very funny to me that we are talking about data transfer (QUIC) but that what happens at "level 8" of the OSI stack can also be complicated depending on what system level 8 runs.

There is no Level 8 as such. Everything happens at the application level i.e Level 7.

Level 8 is a joking way of designating the user. Also refered as "the layer between the keyboard and the screen".

Between keyboard and chair, surely?

PEBCAK, "Problem exists between chair and keyboard"

I think they meant the person and who they are as "level 8".

In other words, if you have this communication protocol, when it reaches level 8 the message might not be delivered or be misinterpreted based on the person. For example in this thread with some people seeing this as racist, some people see it as nothing.

Because those people have a racist mental model of the world.

It is the Baby Yeet meme/Woman Throws a Baby comic by Joan Cornellà.

https://knowyourmeme.com/memes/baby-yeet

That stood out to me as well. I am familiar with the meme, but it is pretty jarring without that prior knowledge. If one of my reports published this under their name we would be having a few words.

Give it a rest. Give us a rest. Don't use tech forums to announce your absurd racial virtue-signaling.

I didn't know fox news had a HN account.

"ThEy DoN't sHaRe My SeNtImEnTs So tHeY mUsT Be A RiGhT-WiNgEr."

Also don't use tech forums to sling around petty insults.

Reply on news.ycombinator.com