The ease of discoverability of a protocol should not be underrated.
I guess if you've never had that moment where it all clicked for you, just by looking at traffic, the significance of human-readable protocols is easy to underestimate.
But hey, Google says that with binary-compressed headers of sorts, they can fit all their tracking cookies in one TCP-packet and that means you wont feel inconvenienced by the tracking payload they are throwing at you everywhere you go on the internet.
You know what? I want that huge bag of tracking cookies to hurt. If you violate my privacy, that should come at a cost.
Google: You can take your SPDY and keep nice, old HTTP alone.
Length-prefixed fields and direct encoding of binary data go a long way to simplifying implementing a protocol.
The ease of implementation of a simple binary protocol should not be underrated.
Complex + non-discoverable = bad engineering
I wonder if that was the original purpose of the protocol;)
The fact is, on the application layer, text based protocols wiped the floor with binary protocols. The answer is in the little details, which everyone knows is where the devil spends its time. They are not easier to implement, because they're more difficult to debug (and because text parsers are, let's be honest, a solved problem); they are not as readable, because debug tooling is never perfectly implemented and never ever present in every system; they are more efficient, but at this layer the efficiency does not pay off (i.e. the size of the HTTP header leading that 5MB image does not really matter).
One of the comments here pulled a comparison I did not remember any more: SS7 vs SIP. Go and have a look at both protocols. They are both mature, so you don't really need to wait for the fantastic debug tooling to appear. Study both ecosystems and then form your opinion on binary vs text application protocols.
Do you remember the OSI binary protocols as designed back in the day?
ASN.1 as far as the eye could see. OIDs. MIBs. Loosely defined extensibility. Terrible, complex protocol designs.
This had nothing to do with the protocols being binary or text; the protocols themselves were painfully complex, making it difficult and frustrating to write a working implementation, much less a complete and interoperating implementations.
> ... at this layer the efficiency does not pay off (i.e. the size of the HTTP header leading that 5MB image does not really matter).
"I don't care about RTT time, the lack of bidirectional communication, and the inability to inline binary data in the protocol stream" ... said no mobile/wireless/desktop developer, ever.
> One of the comments here pulled a comparison I did not remember any more: SS7 vs SIP.
I've implemented SIP. It was a massive hassle because it was text-based and annoying and difficult to parse. The available SIP libraries tend to be buggy and incomplete, and people actually pay for workable SIP stacks. I'd rather not look at SIP ever again.
That isn't to say SS7 is better, it doesn't appear to be. As far as I can tell, that's because it was badly designed, not because of a flaw inherent in using binary encodings.
+1
http://www.igvita.com/slides/2013/breaking-1s-mobile-barrier... if you want to read about this from a mobile perspective.
Now we build a different tool, let's call it http2net. With HTTP being so prevalent, perhaps it isn't unreasonable to think it becomes part of most OS distributions.
So now you are using a tool (http2net) to view/interact with the new protocol and this is good because it is easy to read and use.
In summary, is binary vs text really the big problem here?
You'd think so, but while it's true we can mostly pretend telnet gives you a raw TCP connection, telnet intercepts some character sequences. It just doesn't interfere much with plain text.
My problem with the hypothetical "http2net" tool is that to this day I still have to deal with systems that doesn't have basic tools like tcpdump, curl or wget. But telnet, nc or some other way of getting a semi-raw tcp connection is pretty much always available.
So I have every expectation that it'd take 30+ years before this "http2net" tool would be available everywhere I'd want it.
Regardless of the issue of negotiated options, most telnet clients goes into a command mode if you enter an escape character, defaulting to ctrl+], unless you explicitly change it or tell it not to with a command line switch. This happens for me regardless of port.
tcpdump and strace also confirms what I thought I knew, namely that it also does LF => CR+LF conversion also when connecting to other ports.
EDIT: Here's a simple demonstration of how it is decidedly not 8-bit clean:
echo -e "test\035help" | telnet www.google.com 80
Confirm with tcpdump, or this way if you have strace installed:
echo -e "test\035help" | strace telnet www.google.com 80 2>&1 | grep sendto
echo -e "test\035help" | telnet -E www.google.com 80
(I was, in fact referring to telnet option negotiation which will not happen if not using the telnet port. Since the escape character for interactive use is so easily remedied, it did not occur to me to consider it a problem.)
Engineering should be about taking complex things and making them simple and easy (the best engineers do this), on the other hand you have engineers that try to add complexity because they understand something and over engineer it to obscure it on some premature optimization or cool factor. Taking something simple and making it more complex is the epitome of bad engineering.
Cool is useful and useful is simple or at least simple parts. Keep entry to technology simple, just like good games are simple to start, deeper in it gets more difficult. The door should be easy to enter even though the labyrinth might be immense.
The text-based aspect isn't bad in and of itself. It's the crazy moronic rules and pointlessly flexible syntax that makes it bad. It's the fact that "text-based" is often taken to mean "should allow humans to be flexible in writing it", instead of "uses ASCII". Oh, and UTF8 if you're lucky. If you dare want a non-ASCII value that needs to go into a header, you're toast.
HTTP: Comments inside headers, line folding, context-sensitive header value parsing, and on and on. There is zero legitimate need for these things, yet RFC writers cannot seem to help themselves and design the most complicated syntax as possible.
Text-based formats frequently trick people into thinking they're right because it looks easy on the surface to get something to work. Shit, most HTTP clients/servers aren't actually fully compliant.
It opens more possibility for security holes due to potentially ambiguous parsing (where one implementation misses an edge case, but another doesn't). Protocols are not programming languages. They don't need flexible syntax.
There's also crap left over from a time when people read/write these protocols by hand. Idiocy like the IETF preferred date formats "Sun, 06 Nov 1994". Really? Including "Sun", the English 3-letter abbreviation, in a protocol? How is that useful?
HTTP is being used more and more for high-performance work. Take a look at a high-perf text-based parser, and you see all sorts of ugly hacks. Like doing bitwise comparisons on word-sized integers to determine the request method. (Both my personal code and nginx ended up with similar solutions, so it's safe to say it's a common approach for perf.)
Having a proper binary format that can be quickly, unambiguously, and safely parsed is a huge boon for interoperability and performance. The slight detriment for analyzing raw bytes (which, when encrypted makes it all moot) is not worth it. For a popular protocol like HTTP, you're gonna have plenty of tools to properly parse and analyze.
IP, UDP and TCP aren't text based, and I have no problems regularly analyzing them, nor do they seem to have adoption issues. But you can bet your ass if UDP specified port numbers as a flexible text field, you'd find all sorts of fun bugs and implementation issues.
Edit: After skimming the HTTP 2.0 spec, implementing this as a text-based protocol sounds like a nightmare with no benefit. It's not like you're going to write multiple streams out by hand or something (and use some terrible multipart-mime approach, or another "fun" ASCII-delimiting-binary thing).
Or, as the langsec crowd claims, protocols don't need flexible syntax, lest they actually become programming languages.
You can have simple text protocols and complex binary protocols, but it's more tempting to make text protocols more complex.
(In this comment, simple/complex is in terms of syntax only)
Though I don't really agree with you that you need a binary format for interop and performance. You'd get the same benefits if people just constrained the text protocols. E.g. Your performance "hack" for HTTP request methods is only a hack because method names can be variable length. We could constrain method names to 4 bytes and make the whole thing cleaner while still getting the text protocol benefits of being able to manually inspect request/responses. Similarly, picking a sane date format like iso-8601 and restricting it further to a specific set of options would do wonders while still being readable/writable for humans.
I do agree with you that once a protocol gets as widespread as HTTP most of the benefit is lost as we get better tools to use anyway, though. Especially with debug tools built into most browsers these days.
The issue is not that non-text protocols are impossible to analyse, but that each new non-text protocol requires new tools to do so. For IP, UDP and TCP we've had decades to create good tools. If a common binary serialisation format was agreed and most protocols stuck to that, it would be much less of a big deal to ditch text-only protocols and rely on client side tools to allow reading or manipulating them as text.
I think it's sad that HTTP/2.0 is primarily just trying to multiplex TCP and the rest seems like pure micro-optimizations (the one exception being server push; I haven't read a lot on that yet). It seems like the wrong layer in which to attack the problem, and is basically a huge 'meh'.