>If this takes off, it would remove one of the last few holdouts of non-HTTP application specific protocols.
IRC and FTP come to mind, but yes. It's as if email's being attacked at all angles, lately.
>While using HTTP is not necessarily a bad thing, it's certainly an interesting indication of how application developers prefer convention based protocol definitions (JSON over HTTP) instead of having a strictly defined protocol over a duplex stream (IMAP over TCP).
Using HTTP is a bad thing, I think, because it's a complex protocol and it's entirely unnecessary; JSON is a poor, ad-hoc format that isn't optimal for parsing. I don't believe developers ''prefer'' any of this so much as it's being forced on them.
You can really only define things rigidly if you want to make much in the way of guarantees. Any system held together by conventions can have those conventions violated and this is a harrowing sign of things to come.
They're niche now, though.
HTTP and email protocols are probably the only open protocols that almost everyone uses.
There's ssh, but it's also arguably niche in comparison.
IRC and FTP have widely used secure counterparts. Meanwhile, STARTTLS did not gain the traction that everyone hoped it would.
Edit: Additionally, we have secure working replacement protocols for IRC and FTP (such as Matrix and SSH/SCP).
The problem with this is that most new protocols (or APIs as they're popularly known now) are built by startups aiming for rapid growth. Writing an RFC for the underlying protocol of the newfangled dating app you're building is counterproductive if you don't know the company will exist six months later.
Email parsing can break in so many subtle and/or obscure ways that I have trouble putting any confidence in the parsing code I’ve written… it’s quite frustrating, especially since there aren’t well tested mail parsing libraries for every language/platform to fall back on.
On the other hand, JSON parsing errors are reasonably bounded and there’s well-tested highly optimized libraries for it for practically every supported platform and language. No it’s not perfect or even well suited for the task, but it’s still an improvement.
> If it has to be a loose bag of value data format you could at least CBOR instead of JSON.
so you think there are well tested, good performing CBOR parsing libraries in basically all languages?
I get JSON. People don't want to write text-parsers. But what's attractive about this abuse of HTTP? This isn't hypertext.
Is it so complicated to work with a duplex-stream? Could a library alleviate the complications?
Could it be that developers are more familiar with HTTP than with TCP and so TCP scares them even though it's simpler to use?
> the encoding/transport part of JMAP [i.e. HTTP] is not core to its operation, so future specifications could easily add alternatives
So HTTP is useless for this except for the fact that everyone uses it.
EDIT: actually, one point: you can talk JMAP over HTTP(S) directly from a browser without a translating server.
Ubiquiti is a feature. Not having to explain the caching model to people is a feature.
Email has the other unfortunate property that it has arbitrary binary data (message bodies) mixed in with a UTF-8 control stream, which makes SMTP/IMAP/POP/NNTP especially annoying to implement in languages that draw very strong distinctions between binary and text strings, since you can't just wrap the TCP stream in a UTF-8 decoder/encoder and be done with it.
It's not that duplex-stream is complicated, it's just that request/response semantics are much more suited for most use cases. Where duplex streams make sense, we now have websockets. Even then, on top of that you'll end up implementing stateful request/response semantics because you need some structure unless you're literally dumping binary data into the socket.
That ship has long sailed. HTTP is no longer just hypertext -- it's a ubiquitous and well-understood transport for RPCs, with lots of tooling and infrastructure support across the Internet.
I think at its core, HTTP abstracts away some of what you would want in a request-response protocol, but not really all of it. But _simply because of the ubiquity of tooling_, you get a whole lot of other stuff for free (load balancing, DDoS detection, in-browser debugging, etc.)
> Is it so complicated to work with a duplex-stream? Could a library alleviate the complications?
Yes, TCP streams are very complicated to work with (in my experience) -- there are way too many failure modes and corner cases to consider. IMO, your language's go-to HTTP library is that library that alleviates the complications.
All this said, I personally think gRPC is the right approach for RPCs, and I hope to see broader adoption of it over the coming years.
Isn't it the same thing, but simpler and more flexible? You write a message, and then you read a reply, rinse and repeat. Could you expand on such things that are complicated with TCP that HTTP helps with?
well, no, you don't "read a message" with TCP, because it is a stream-oriented protocol. you read bytes that keep coming and wait until you have enough or reach a delimiter that you chose.
I'm getting the feeling that the popularity of JSON and HTTP come by good part from the lack of good parsing libraries in many languages. People generally only know how to work with what regexes are able to handle and nothing more.
Even if you have good parsing libraries, parser combinators, etc, you still need to write the grammar - which can be hard if you are not accustomed to that... years after I still find an occasional bug in grammars I've written in the past. Compare this to JSON over HTTP which ends up basically being
var xhr = new XMLHttpRequest();
xhr.open('GET', "https://my/api.json", true);
xhr.onreadystatechange = function(e) { var response = JSON.parse(xhr.responseText); }
To be clear, I'm not saying that there's no place for socket programming -- there definitely is. I've spent a good part of my life doing obscure network programming on all kinds of platforms, so I try not to use TCP/UDP unless I have a really good reason. (It's kinda the same reason I won't use C or C++ -- too much to worry about for most general purposes.)
> connection setup and teardown
If there's no benefit in avoiding setting up and tearing down a connection with each request like one typically does with HTTP, then I can see how it might be an annoyance compared to making a single function call from a library that makes the request and returns the response.
> buffering
A parser should deal with that. It could be JSON parser.
> serialization
That's still a problem with HTTP. People typically use JSON for this and it doesn't depend on HTTP.
> byte ordering
parser/JSON's job.
> your own status/error handling, cache control, content encoding
That can be beneficial or it can be problematic. If you need such things, then yeah, it's cool to not have to think about making your own. However, if you don't or if your needs don't align with what HTTP provides, that can be problematic. For example, from a cursory search I see the following in the JMAP spec:
> Implementors must take care to avoid inappropriate caching of the session object at the HTTP layer. Since the client should only refetch when it detects there is a change (via the sessionState property of an API response), it is RECOMMENDED to disable HTTP caching altogether, for example by setting `Cache-Control: no-cache, no-store, must-revalidate` on the response.
So, HTTP provides caching, but not the caching mechanism that's needed, so it's recommended to explicitly disable it completely.
With respect to status and error handling, it seems JMAP still had to define their own errors to be described in JSON[1], so not a perfect fit there either.
On the content encoding, I'm not sure if JMAP would use something other than application/json, so it seems like something that wouldn't need to be specified if one didn't use HTTP.
I know the discussion is not really about JMAP. I'm just using JMAP as an example on the above.
> TLS etc. (Simply, doing TLS correctly itself is not easy.)
I would hope that there's a library equivalent of the shell's:
openssl s_client -connect $host:$port
[1] https://jmap.io/spec-core.html#initial-jmap-error-codes-regi...
2. I've heard about corporate firewalls which close everything and nobody has power to overcome that, so I'll tunnel everything via HTTP(S) and forget about those horrors.
3. HTTPS provide absolutely hassle-free encryption with reverse-proxies. I don't even need to think about it. Using TLS library sounds too complex and I must make some real effort to implement it. Rolling out my own encryption? Everyone will scream!
4. HTTP/1.1 protocol is really simple. There are some hidden gotchas in the corners, but basically it's headers, empty line and then either bytes or stream of chunks, that's all about it. So I can pass routing information (URL path), I can pass some metadata in headers and I can pass either a singular message or a streamed content. And with HTTP libraries in every language you'll get that functionality almost for free, both for server and client.
5. Theoretically you can leverage HTTP caching on various levels. Might be tricky, but certainly easier rather than implementing it from the scratch. And with things like Cloudflare you can't really build anything similar.
6. Browsers can only talk HTTP. So even if I don't need it know, if I would ever need to build web application, I'll have to provide HTTP bridge. And web applications are really common nowadays. You can't really make IMAP client in JavaScript running in browser. But you probably can do it with JMAP.
Also HTTP carries some overhead. It's not significant, something like 50-100 bytes, but it might matter for a lot of tiny messages, I guess.
Another factor is: those intermediate proxies might break your app. They might cache your responses even if you don't want that, so your application will act wierdly. Of course any intermediate proxy might break any protocol...
My (weak) understanding here is that in most mobile cases this is intrinsically problematic and is the reason for the plethora of push notification standards. In this case HTTP is more battery friendly as a noisy server can't as easily spam your phone with traffic.
I mean, if you need to spam your phone with traffic, you'll do that. If you're writing whatsapp, you want to deliver messages instantly if user has app opened, so you'll keep TCP connection to the server.
Slightly different than ubiquity of tooling is firewall policies: HTTP(S) is the protocol most likely to get where it needs to without someone configuring something to allow it, so tunnelling everything over HTTP(S) is the path of least resistance to widespread use.
Plain TCP essentially emulates a full duplex serial port. Applications need more on top of that. Methods for framing, attaching Metadata, data flows, secure transport layers, etc.
So HTTP which provides a modicum of those options ends up being default, because it's available.
For comparison, OSI defined various components to handle those elements and a proper OSI stack should have then available to application developer.
Another example would be Symbolics Lisp Machines (Genera) where the network layer provided a bunch of mid-layer protocols, like one which provided messages that were formed of basic typed data structures. So people would use those to quickly build custom protocols for their applications, thanks to not having to deal with low level details.
Various Middleware also come to mind.
TCP is not enough.
HTTP is a botch that is slowly getting better and is far too heavyweight for most things.
There’s lots of stuff that HTTP isn’t eating.