138 karma · joined January 4, 2020
Work on QUIC protocol and other networking things at Google. Long time ago I was quite active in various open source projects, notably MediaWiki.
All opinions are my own, and do not reflect the official position of my employer or any standards organization I'm involved in.
This is somewhat funny to read because this specific issue in CUBIC (sudden CWND jump upon existing quiescence) was originally discovered in Google's QUIC library and then later reported to the team working on the TCP stack. I know this because I was the one who found that bug back in 2015.
That said, congestion control algorithms are really prone to logic bugs, and very subtle changes in the algorithm can often lead to dramatically different outcomes. Because of that, there's a lot of value in running congestion control code that has been tested on a wide variety of real Internet traffic.
[0] https://chromium.googlesource.com/chromium/src/+/HEAD/base/m...
> In addition to the low roundtrip time, the connections between your load balancer and application server likely have a very long lifetime, hence don’t suffer from TCP slow start as much, and that’s assuming your operating system hasn’t been tuned to disable slow start entirely, which is very common on servers.
A single HTTP/1.1 connection can only process one request at a time (unless you attempt HTTP pipelining), so if you have N persistent TCP connections to the backend, you can only handle N concurrent requests. Since all of those connections are long-lived and are sending at the same time, if you make N very large, you will eventually run into TCP congestion control convergence issues.
Also, I don't understand why the author believes HTTP/2 is less debuggable than HTTP/1; curl and Wireshark work equally well with both.
Both JSON (as defined in the RFC) and JSON5 have a nice property of being well-defined, meaning that you can use different libraries in different languages on different platforms to parse them, and expect the same result. "JSON but parser behaves reasonably (as defined by the speaker)" does not have this property.
The most up-to-date version of HTTP/1.1 spec is RFC 9112, which says:
> Although the line terminator for the start-line and fields is the sequence CRLF, a recipient MAY recognize a single LF as a line terminator and ignore any preceding CR.
"MAY", of course, is different from "MUST" or "SHOULD", so I feel like the author's claim that implementations rejecting bare NLs are broken is at odds with the specification.
[0] https://www.w3.org/policies/patent-policy/#sec-Requirements
Error correction codes on L4 level are generally only useful for very low latency situations, since if you can wait for one RTT, you can just have the original sender retransmit the exact packets that got lost, which is inherently more efficient than any ECC.
(full disclosure: I worked on some of this stuff)
[0] https://datatracker.ietf.org/doc/html/rfc9000#name-address-v...
[1] https://www.fastly.com/blog/quic-handshake-tls-compression-c...
I feel like this is partially because the main value of the API comes from working better on lower-quality networks, rather than providing the ability to do something completely brand new.
> FF and Safari don't support it yet.
For what it's worth, Mozilla are positive about it [0] and are actively involved in the standards process. I don't believe the WebKit team has stated their position, though there are Apple people who are involved in the standards process too.
> I think it's missing a killer app. But I am not sure what WebTransport would enable that isn't already possible with current APIs.
When we started out working on WebTransport, there were two major use cases we had in mind: live media and web games. IETF MoQ working group [1] will hopefully address the first one. I don't know whether anyone is using it for building games, but I don't actually have that much visibility into what's going on in that space.
[0] https://mozilla.github.io/standards-positions/#webtransport [1] https://datatracker.ietf.org/wg/moq/about/
If you want an example of an application that actively benefits from using WebTransport, there's a proposed video streaming protocol called WARP: https://datatracker.ietf.org/doc/draft-lcurley-warp/
The lack of good examples mostly stems from the fact that there isn't really that many publicly available server libraries yet. The two I can name off the top of my head are aioquic (Python) and the Google QUIC implementation used in Chromium and ENvoy -- the latter is sadly not trivially embeddable by third-party code (I work on it, and I've been trying to make it easier to use, but given that it's a large C++ codebase with a lot of dependencies, this has been taking a while).
For YouTube, the CPU cost of QUIC is comparable to TCP, though we did spent years optimizing it. [0] has a nice deep dive.
Other CDN vendors like Fastly seem to have the similar experience [1].
I believe sendmmsg combined with UDP GSO (as discussed, for instance, in [2]) has solved most of the problems that were caused by the absence of features like TSO. From what I understand, most of the benefit of TSO comes not from the hardware acceleration, but rather from the fact that it processes multiple packet as one for most of the transmission code path, meaning that all per-packet operations are only invoked once per chunk (as opposed to once per individual IP packet sent).
[0] https://atscaleconference.com/videos/networking-scale-2019-i...
[1] https://www.fastly.com/blog/measuring-quic-vs-tcp-computatio...
[2] https://blog.cloudflare.com/accelerating-udp-packet-transmis...
(FWIW, you can send RTP over WebTransport datagrams, instead of using SRTP and ICE-lite)
If anyone is wondering if this is already implemented anywhere, we're currently experimenting with it in Chrome: https://web.dev/quictransport/ -- I'd be curious to hear what people think about it.
The best approach is typically put a length in front of every message. The good things about that approach are:
1. The receiver can allocate buffer that is exactly the size it needs to fit the message. 2. The receiver can check whether the message is too long before seeing the entire message.
The only disadvantage is that you have to know the length of all messages in advance.
There are algorithms that try to use increased delay as a signal that the link is full. This approach has multiple problems, one of which is that delay can be really noisy on wireless networks; another is that if you have a loss-based and a delay-based connection sharing the same link, the delay-based one will get much less than a fair share of its bandwidth. People have been trying to make an algorithm that both coexists with Reno/CUBIC and does not induce bufferbloat for the last 25 years or so, and there's been some progress, but none of it has reached the point where it could be used as a default congestion control for all operating systems.
The problem of "I have files to transfer in background, but I want my connection to yield to more important traffic" can actually solved using a special congestion control algorithm called LEDBAT [1]; it's used by Apple for things like software updates, and BitTorrent uses it too. Unfortunately, I think only Apple implements it in its TCP stack, so anyone who wants to do that would have to roll their own thing using UDP.
It's a "reasonable default", in the sense that it normally works quite well, and is easy to implement. The dangerous thing about congestion control is that it's really easy to get it wrong and not notice (as noticing requires performance testing and monitoring).
Google QUIC uses BBR on the server side. I wrote the initial version of said BBR implementation, and it took a lot of effort to get it to a production-ready state, so I would not generally recommend writing BBR from scratch. CUBIC is much simpler than BBR, but is also prone to subtle bugs like [0].
> 2. Not everything that needs encryption needs TLS specifically. TLS is not a one-size-fits-all solution.
I'm actually curious what use cases you have in mind. TLS definitely does not cover 100% of all possible cases, but I've seen a surprising number of cases where people rolled their own thing when using (D)TLS would have just worked.
[0] http://bitsup.blogspot.com/2015/09/thanks-google-tcp-team-fo...
Somewhat tangentially related, for cases when P2P connectivity is not needed, there's also a proposal that lets you send data unreliably, but in a client-server setting: https://github.com/WICG/web-transport/blob/master/explainer....
Both are currently being actively prototyped in Chromium (full disclosure: I've been working on the latter).