Multiplexing on top of TCP (2011)
250bpm.com
250bpm.com
Google is aware of that issue, that's why they explored SCTP first and now QUIC as alternate layer 3 protocols to replace TCP in the future.
QUIC, which runs over UDP, is easier to adopt. However, in reality many ISPs and company networks de-prioritize UDP traffic so the quality of connection is unpredictable.
TCP is still the safest bet. We need to figure out a better (and magical!) way somehow...
The idea that middle boxes need to be aware of protocols is incredibly destructive to the end-to-end principal - the critical property of IP networks that enables a rapid pace of permissionless innovation. Middleboxes are evil incarnate.
Fortunately, academic research as well as operational experience has shown that TCP and UDP provide the primitives required to build any protocol. QUIC, which you mentioned, is an example of this. Google has made the explicit - and probably wise - decision to keep protocol state encrypted. While there would be some nice optimizations possible if routers could be aware of things like 'this session is closed now', the risk of middleboxes making it so that the protocol can't continue to evolve is too great. The solution is to specifically prevent middleboxes from having the knowledge needed meddle in an 'intelligent' way.
The last statistics I saw from Google were that 95% of networks don't cause problems for QUIC (i.e. QUIC connections are as fast or faster than HTTP/2). The main issue I'm aware of is not so actually de-prioritization but rather rate limiting (it's often implemented as a quick - and very very dirty - DDoS mitigation technique). Google's stated solution is to fallback to HTTP/2 based on measurements of user experience on a per-AS basis.
Networks which try to be "intelligent" rather than just provide adequate bandwidth will tend towards providing a crappy user experience. Solutions which optimize for well behaved adequately provisioned networks and have an acceptable-though-degraded fallback seems like a reasonable way to promote progress.
SCTP has an RFC-defined UDP encapsulation which will deal with many evil networks.
That's... great, and I'm glad there's progress being made somewhere, but I don't think it's fair to say it's "the present". It's on its way.
My ISP has rolled out FttH across a wide area but still only offers IPv6 on business buildouts. Even if they offered IPv6 to general customers, they still have a ton of home gateway devices out there that don't support any IPv6 at all. The other ISPs don't offer it or have any external roadmap either. I don't think this situation is all that unusual or unique - it's just not something that the people paying the bills are clamouring for and it's a long road from where we are to significant penetration.
Oh, and EC2 still doesn't support IPv6 on anything but their load balancers. What portion of the market and traffic do you think they make up?
Is this true? As far as I can tell SCTP works over plain IP, just like UDP and TCP do.
Many middle boxes, especially cheap home routers, are not aware of IP protocols other than TCP/UDP (and maybe GRE/ESP if you're lukcy). Their NAT will not work with SCTP.
Fortunately, mine hands me a /64 so I'm not in that situation. :)
Not to say I wouldn't want SCTP to succeed but IPv6 doesn't exactly spring to mind as a model for the successful adoption of a new internet protocol.
What makes you think this? I play a UDP-based FPS game where "ping" times are critically important and it seems like latencies across the board have continued to drop over the years.
Latencies are ~2x the speed of light in most case. Not too shabby.
My home ISP throttles UDP traffic to about 5Mbps while TCP reaches 80+Mbps. Packet loss during peak hours can be as high as 60% for UDP but a manageable single digit percent for TCP.
My workplace has anti-BitTorrent mechanism deployed and any meaningful amount of UDP outbound traffic is dropped.
So there's still one connection between the edge and the browser (can't do anything about that), but once it gets to the edge node, each request is sent over a prenegotiated connection to an edge node near the server.. then it drops down to a more reasonable number when sending the requests to the customer's server. So if a packet is dropped, it only affects that one file/connection.
Don't think any other CDNs are doing this yet.. but I think it'll become more common as they invest more resources into their http2 implementations.
At Facebook, we had clients talk http2 to edge nodes near the users, which then edge terminated TLS and opened up the request to figure out which upstream was best to forward the request to. We kept http2 conn pools to each upstream from a given edge location. We also had tunable amounts of domain sharding so that the client could open N http2 conns at once to a given edge hostname (cdn[1-4].facebook.com vs just api.facebook.com or www.facebook.com). This helped with head of line blocking, assuming that your ISP wasn't hopelessly over-congested. In that case, no amount of sharding will help you, and sometimes it even works against you depending on what tcp middleware your mobile ISP is using.
We did different TCP settings for edge<->origin conns than edge<->user which let us balance between latency and throughput for origin traffic as needed, and we did some interesting work tuning edge tcp settings differently for different user networks too. Things like determining the best congestion window for a given ISP's users, how large to set the send buffers, etc.
Admittedly, this work was much easier since we had good http and tcp instrumentation, and we had high quality links between edge and origin. I just wanted people reading your comment to understand the tradeoffs. We saw a measurable difference in perf with a full http2 stack that made up for the extra complexity of running http2 top to bottom.
I should be more specific in my description: We are using http2 everywhere.
There's a single http2 connection between the client and edge (the customer could setup sharding, but there's no way for us to force it).
That edge node has a pool of http2 connections to other PoPs, and maintains QoS stats for each PoP. It'll pick the fastest path to send the request. The pool is large enough to service all of the requests that are being handled at any one time. So in effect, if the client sends 10 requests to the edge node, each of those will be sent edge<->edge over 10 http2 connections. (The connections in the pool are kept active, so there's no tls resumption delay either.)
Once the edge node near the origin has received it, there's a smaller http2 pool to the origin server.
So at the origin, it's handling a handful of requests in parallel.. even if there's only a single client making all of the requests.
I think that's a bit clearer.. I've been happy with the performance we're getting from this.. but plan to test sctp and quic later this year for edge<->edge.
In other words.. he's saying, because HTTP 2 is multiplexed, tcp's head of line blocking becomes a larger problem. Which is true.
It really is unfortunate that we ended up with UDP and TCP as the only choices. There are so many more things you might want outside of "unreliable packets" and "automatically recovered streams."
* Most ISPs don't mangle UDP, and if you care enough about perf to be using UDP in the first place, you can afford to figure out who mangles UDP and fall back to TCP for those networks. It's a very small number of networks.
* Any TCP/SCTP/etc inspired protocol you can think of can be implemented on top of UDP. Its header is only eight bytes long and it adds minimal overhead. UDP socket semantics are simple enough that you can treat them as a slightly smarter IP socket and build your protocol as if you were running directly on top of IPv6.
* There's a huge advantage to being able to implement your protocol in userland since you get fine grained control over congestion and connection logic. There's a huge disadvantage too, since it's actually hard to implement that stuff yourself. But for the people who are large enough to build their own protocols (FB, Google, etc), it's worth it.
* You can even encapsulate SCTP inside UDP to punch through middleware firewalls if you want to. I don't know of anyone doing that for real in the wild, but there's a RFC for it (RFC 6951).
The fine grained control over congestion and congestion logic is cool, sure. But more important in my mind is the fact that there's a cross-platform API available and you can rapidly make changes to how the protocol works without needing to require a specific kernel version or a specific operating system.
There's a whole lot of really cool frameworks for SDN to allow people to work at lower levels, but that's the problem - a standard API that everyone can expect to be widely available hasn't really emerged yet.
Aren't WebRTC data streams implemented with SCTP over UDP?
E.g. "ICE" provides a negotiation protocol to make a couple arbitrary machines able to talk to each other directly, including the NAT hole punching part of things. In the limited % of cases where there's no known way to punch a hole through NATs, an automatic fallback (using a relay server) is provided as a part of the protocol.
It's not elegant, at least we don't have everybody re-inventing the wheel. And, good results when ICE is used can become part of the criteria now used to judge the quality of NATs and other evil middlebox devices.
Anyway, I'd love to be able to pick and chose among:
1. Packets versus streams.
2. Reliable versus unreliable.
3. Immediate delivery, or delivery in the same order as data was sent.
Right now, we have UDP, which is packets, unreliable, immediate, and TCP, which is streams, reliable, delivery in order of transmission. But all eight possible combinations make sense (albeit some more than others), and it's unfortunate that we have to reimplement stuff that already exists if we don't fit within the two choices we have.
And if we're going beyond what TCP provides, I'd really like to see:
4. Choice of ACK versus NACK based loss notification.
5. Forward error correction.
6. Multihoming.
#5 should really be done at the framing layer, otherwise your frame delimiters can also be corrupted. It's also very tied to physical layer and probably better off not being configured in platform agnostic software.
No opinion on #6, I've not deal with it much AFAIK.
I also missed one, which is hard to get right: rate limiting. Especially if you want to coexist with TCP rather than drown it out (or be drowned out by it).
There are also (low-traffic) use cases where overrun of receivers is unexpected so you don't need an application-level ACK mechanism, just notification of overrun so you can handle it, e.g. by breaking up and restarting the application associated with the overflowing stream.
So basically the only thing the HTTP/2 crowd could come up with as a real advantage over HTTP/1.1 (discounting ofcourse that HTTP/1.1 supports pipelining), which is a massive source of complexity, which will also be a massive source of bugs, does not provide the benefits it claimed it would.
Can we just call off HTTP/2 yet? It was pushed by Google to cover their needs and agenda, without any respect for the protocol's history. It had a bunch of riders introduced with misleading language with regard to privacy violating semantic changes (For instance HTTP/2 cannot do regular HTTP, only HTTPS. You can no longer deploy private apps on your LAN without registering with centralized internet registries such as DNS and CAs. Etc etc).
Basically HTTP/2 was Google's attempt at making it easier for them to keep their huge amount of tracking cookies on every single HTTP(S) request across the internet, without it causing too much of an impact on users.
You know what? If I have 200KBs of Google-cookies tracking me, I want those cookies to impact performance. I want to know something is up.
Can't we just say "HTTP/2 considered harmful" at this point?
I'll be disabling it in all my browsers which support proper user-managed configuration. Needless to say, that excludes Chrome.
HTTPS-only is NOT a protocol limitation. It was pushed by BOTH Google AND Mozilla, in the name of security, like all the new features coming to the web (need webrtc ? Https. Want the camera ? https. Want https ? https). I'm not actually too sure I like that trend either, it feels like shoving candies down my throat. I like candies. But not like that.
However, saying this is a Big Google Conspiracy is ridiculous. I mean what the hell does cookie have to do with anything.
PS: On a separate note, could we get 10.0.0.X, 192.168.X.X, 127.0.0.1 and the file:// protocol counted as "Secure" please ? It's a pain to have to create a self-signed certs just to develop stuff.
And let's not mention the fact you can get a free HTTPS with let's encrypt, MITM an existing 10.0.0.X connection and serve a 301 to your https-enabled domain name.
My use-case is rather simple : I host a few things at home, and I've had to install a root CA on every device just because I can't do webrtc otherwise.
I could get a letsencrypt certificate for a domain name, but honestly that sucks.
I believe in edge it needs some kind of registry setting.
Then you complain that you can't use HTTP/2 for some potentially useful cases.
Then you proceed to accuse Google of pushing HTTP/2 for improving the performance of their websites, effectively conceding that HTTP/2 does improve performance of multi-resource webpages.
Then you argue that it's actually a good thing when a multi-resource webpage is slow.
HTTP/2 does not require HTTPS. But all browsers decided this was a good thing, so browser require HTTP to enable HTTP/2. And you do not need to get a centralized CA to give you certs; just use your own.
HTTP/2 makes parsing way faster, which can save significant resources. It'll also cut down an ambiguity which might prevent some security issues.
By the way, maybe it should say in the title that the post is from 2011?