Tcpls: Modern Transport Services with TCP and TLS
blog.apnic.net
blog.apnic.net
TCPLS seems like a good proposal but I won't hold my breath for it as the big players like Cloudflare, Google, Akamai, Fastly are already committed to QUIC. There's still significant churn in the proposals and some features like MASQUE are still being finalized, but overall QUIC works really well.
It seems likely that a lot of the issues raised in the OP will be somewhat forced to be resolved due to the aforementioned ramrodding of QUIC into the HTTP3 path, but maybe in the long run the two ideas will co-evolve to the same solution.
QUIC took 4 years and 34 drafts to get standardized. The final IETF version is significantly different from Google initially came up with. So, no it most definitely was not ramrodded through.
Considering HTTP/1.1 stood for over 15 years before http/2 came along (and was a pretty modest improvement over http/1.0 compared to what we're talking about now, so I'm being generous by not saying over 20 years). http/2 had only been promoted to a proposed standard for 3 before http/3 was proposed. That, to me, is a very aggressive process. But again, it's the relationship to http that bothers me, not the standardization process for quic itself.
At any rate I also didn't say they shouldn't experiment with using it as an accelerated path for http. I didn't think they shouldn't with spdy either. But historically RFCs were built on practice, not the other way around.
If I were the Internet Dictator, I'd be a big fan of standardizing the semantics of http separate from the physical transport and then defining best practices for using alternative transports in the wild, and then eventually standardizing what comes out of that process.
To me it's an indictment of how all this shook out that we're so quickly looking to replace http/2. I think moving to quic is a good move but maybe we didn't need an official http/2 standard in the meantime.
Everyone from FB to Microsoft to even Huawei wanted QUIC. It's was not something that only Google was interested in. Hell, the primary author left Google well before the WG was done.
> But historically RFCs were built on practice, not the other way around.
Right from the early drafts, there have been multiple independent QUIC implementations constantly being tested. There was plenty of working code before there was rough consensus.
> standardizing the semantics of http separate from the physical transport
What actual semantics of HTTP changed with QUIC?
This is what the IETF has been doing for several years already as part of the core HTTP specification rewrite. Namely, see the HTTP Semantics document https://www.ietf.org/archive/id/draft-ietf-httpbis-semantics...
Semantics are common, how those map to specific transport - how it looks on the wire - is then the job of each individual HTTP version.
It reads like you're overlooking the act of standardization in improving the definition of a protocol. Experimentation is good but often only focuses on a limited set of use cases, deployments, or implementations. Standardization is great for interoperability across a more diverse set of all those three. This typically yields protocols that are more generally applicable for a wider population of the Internet.
Google QUIC was focused on an HTTP use case and thatbran through the entire design. Through the IETF process, the design evolved to be a far more generic transport with rich extensibility and modularity.
QUIC was "intermingled" because it exactly solved this problem at the transport layer, the only place it can be solved. They couldn't just introduce any random thing, and no other protocol existed, so they ratified QUIC independently as its own protocol and applied it to HTTP/3 at the same time. They had to introduce a new standardized transport if the hope was to solve the HOL blocking issue. If you "dis-entangled" it you'd just have HTTP/2. The new transport protocol is the defining characteristic.
Furthermore it all would have been pointless to define QUIC as a standard without actually applying it in practice, in anger, to real workloads. Which is exactly what multiple stakeholders in the process did throughout the multi-year ratification process. What are they going to do, define QUIC, sit on their thumbs for 5 years and cross their fingers hoping someone else tries it?
There are other factors like most of the involved stakeholders had previous experience watching with other attempted standards in similar veins like TCP Fast Open die on the vine. That also influenced decisions like using UDP for QUIC's transport (and the ever present discussions about middleboxes.) Actual experience indicates most low-level protocols have ossified badly or are handled very poorly, including TCP and UDP except most things can handle those, so moving the transport layer into the application stack is seen as providing actual agility needed to respond to issues like the HOL blocking issue. That's how they revised the protocol several dozen times over the years; applications are actually fairly agile. Now you could argue these people should have discovered that issue when designing HTTP/2, but "Why didn't this thing get discovered and found out?" is always a weird complaint. Why would you expect everything to be found out early? I just can't answer that complaint honestly. Weirder things have happened than "Some engineers missed something."
Many people seem to hold some unwritten belief that if a protocol is defined as, say, an application layer protocol at one point, then it can never ever redefine its parameters to be anything beyond that. It can never become a transport protocol, for example, or have the two be mandated together, despite decades of experience suggesting that might be a fruitful path for many technical reasons. But there's no random rules of the universe defining what a network protocol looks like. It can be whatever we want it to be. Humans define this, there isn't like a book of 10 commandments out there written by God.
There is also no necessary indication that HTTP 1.1 sticking around for so long versus HTTP/2 necessarily meant it was the best among all options, by the way. We abused it a lot for many things. Being around for so long could also mean we were just bad at enacting change; certainly browser monocultures have made change easier for better and worse, on that note. But the whole HTTP/2 and HTTP/3 process has been about 13 years on its own if you count it as starting with SPDY. It's not like there's 200 years of precedent for how long it should take.
> They had to introduce a new standardized transport if the hope was to solve the HOL blocking issue.
I take no issue with quic being standardized and I applaud that it was.
> Furthermore it all would have been pointless to define QUIC as a standard without actually applying it in practice, in anger, to real workloads.
Protocols are deployed on real workloads in anger in practice every day without even having an ietf rfc, let alone being blessed as the next official version of one of the most important protocols on the internet, and it works out fine. I never said, meant, or in any way advocated for anyone sitting on their ass.
EDIT: I was too rude/curt and re-edited this. Sorry.
1. Many of the people leading development of the older HTTP versions were academics.
2. See, e.g., CurveCP: https://curvecp.org Having played around with CurveCP before QUIC existed, and having already thought CurveCP was amazing, the later commercial effort led by Google has left me underwhelmed.
Compare that to finalizing a new protocol, putting it in the kernel, and letting all applications just connect() and send() and recv(). We lose the ability to constantly mutate the protocol whenever FAANG feels like it, but we also gain the ability for every application to use the new protocol just by changing a flag passed to a syscall.
The real issue is that we can't get away from UDP/TCP because of middleboxes (firewalls, load balancers, etc) and the fact that switches only know how to hash on 5 tuples of protocols they know about. If you're lucky, your router will be able to look inside a GRE header or something, but that's about it.
When was the last time you used SCTP, for example? (If you've used it recently, I bet it was tunneled over UDP)
High end routers are able to do pretty conplex lookups into their asics, but this usually only extends to specific use cases. (as far as I am aware for instance, an juniper MX series is able to do a lookup of both vlan tags in a single asic pipeline step when using q-in-q. the same also applies to mpls stacked lable lookups on provider edge routers).
I'm not sure the distinction you're drawing here. When my switch balances based on port numbers,-- yes, it's technically just peeking at "data within the ethernet frame". (is this IP? OK, let's add the address at this offset to the hash. Oh, it's TCP too? let's add the port numbers at this offset to the hash).
L2 hashing puts everything from a single pair of hosts on the subnet on one wire, but this can suck when a lot of traffic comes in through a gateway. L3 hashing fixes the gateway problem. L4 hashing lets individual connections have entire links. And, of course, the more fine-grained the hashing, the better the balancing can be expected to be and the lower the odds that 80% of your traffic goes down 1 link in a 4 connection bundle.
This whole middlebox obsession is some kind of Stockholm syndrome. Here you are telling me there's people who make shitty devices that don't respect network traffic and because of that you can't change network traffic. It's like saying there's someone breaking into houses to steal stuff, so we can't install stained glass windows because a thief might break them.
Unfortunately it's over 20 years old and people just shove it in user space and use it over UDP (WebRTC).
Your sentiment is appreciated but it wouldn't be a way to get anything actually done :/.
This contrasts to IPsec transport mode which only encrypts the payload not the IP info https://upload.wikimedia.org/wikipedia/commons/6/64/Ipsec-es...
Generally there is encryption available at each layer in the network stack but middleware (operating systems, firewalls, NAT) tend to get in the way at lower layers. E.g. IPsec is often now run with NAT traversal which means the ESP header is stuck under a UDP header so it can get through to end devices even though IPsec itself doesn't require this. This, and a few other things, have left IPsec as a thing most people only know for corporate VPNs.
Switching to IPvX would be the same problem as convincing people to always encrypt with IPv6 just amplified as there is 0 hardware with existing support for such a protocol and IPv6 has had a 20 years head start on rollout that would be wiped away. It'd have the added "bonus" of being even harder to pitch adoption for since "NAT on IPv4 is working fine" and you'd still have to wait for those people THEN the IPv6 people to migrate.
Meanwhile things like QUIC, HTTP/3, and TCPLS give all the same advantages as migrating to the hypothetical IPvX without breaking when they run into middleboxes. Just think of it as IPvX but with a wider header field than you were planning. After all you'll still need some header that tells the network where to deliver the packet then the encrypted payload it's just that that dumb destination header uses more bits than it needs to if it were designed fresh.
i think a major design goal of something like IPvX would be to eliminate the need for any firewalls, NATs or network partitions. everything would be hardened for the public internet, every datagram would be encrypted, every listening service would be ddos resistant and this would all be handled by the networking stack by default.
crucially i think that there would have to be a hook to get end users to use it. i think there's a lot of room for next generation decentralized public internet identity, naming, reputation and commerce protocols. if these next generation services only lived on IPvX and made direct use of expanded address space, then maybe it would be compelling enough for an upgrade.
of course there are concerns about replacing all the old networking hardware and the environmental impacts of such, so maybe that could be mitigated by exploiting planned sunsets and also mark a sunset for planned sunsets.
spitballing is fun! :)
https://twitter.com/alagoutte/status/1532013841718120449?t=W...
> if you want reliable data transfer over UDP you have to basically implement almost everything TCP does on top of it.
Even QUIC has an ACK-like signal so you will never avoid it as otherwise you will deal with the two generals' problem (https://en.wikipedia.org/wiki/Two_Generals%27_Problem). The reason QUIC is more efficient than TCP is simply due to the fact that TCP can't evolve further due to middleboxes (called ossification, https://en.wikipedia.org/wiki/Protocol_ossification), not that TCP is inherently bad.
Note that your example of file transfer is a special case that an application can be specifically tuned for; it can have a receive window that is as large as the file it is going to receive, because it can use the file on disk as the backing store for that window, and it can implement an unlimited form of selective ACK. This is really not something that you can do with a generic transport protocol that implements the POSIX sockets API.
There are differences in behavior regarding what happens if loss is encountered and what exactly is retransmitted, but those still don't make QUIC radically different in terms of behavior and performance.
There's some good ideas in this one! The idea of just keeping a TCP stream per Tcpls stream while sharing some crypto state means it can be really efficient, since all optimizations that exist for TCP (including hardware offloads) will continue to work. That is also visible in the efficiency benchmarks [1] on the site.
One might even argue that with such a design where the most expensive parts of TLS (handshakes) become no-ops an introduce no additional latency, techniques like HTTP/2 become unnecessary - one could keep just having a single TCP stream per request and enjoy better flow control, fairness and observability than with anything application managed.
However there are some challenges that need to be resolved for running protocols on "real-world infrastructure and datacenters", which change the picture a bit. One is that a single IP address announced by large-scale webservices doesn't necessarily point to a single server, but to e.g. a rack of them or even bigger units. This means that a second TCP connection might (or even should be) established to a different server - which would not be aware about the TLS session state. If the state is kept on a different server, the original one would need to forward all packets either via IP-in-IP encapsulation or by creating a separate upstream TCP connection and proxying all data bidirectionally. All those approaches would be fairly expensive - every time a TLS session would be joined the amount of packets handled by a server would be double of the ideal version.
While a similar challenge exists for QUIC, it's actually cheaper on average: Since multiple streams use the same QUIC connection and usually also the same source/destination IP and port pairs, traffic would usually be directed to the same host by common infrastructure and require no special handling. It would only require effort to "fix" misdirected QUIC connections in the event of connection migrations - which are rather rare. If packet forwarding is required, it can be achieved rather cheap and easily by interpreting the QUIC connection ID fields in all UDP packets and doing raw UDP packet forwarding.
Therefore as far as I understand the proposals so far, it might not be necessarily clear-cut on whether running QUIC or TCPLS would be cheaper and an ingress or load-balancing layer - which are the layers where both protocols have the biggest benefits due to the highest amount of concurrent connections and higher client-facing RTT.
But maybe the challenge for TCPLS could be somehow solved by having a shared place for the TLS state between hosts and making sure streams are operating fully independently - but it requires a bit more thoughts.
I'm wondering how close such a proposal is then to the existing TLS session resumption mechanisms - just extended by allowing more than 1 follow-up session.
[1] Note that the diagrams are really about efficiency and not performance. They tell nothing about what throughput might be available for end-users that are e.g. 100ms away from the server. For local tests the delay and packet introduced by large-scale internet infrastructure doesn't play a role.
But thanks for the correction! I appreciate it.