Building Protocols with HTTP
tools.ietf.org
tools.ietf.org
The thing to realize here is how much middleboxes sit between a client and a server, boxes which are aware of higher layers than just IP. It's impossible today to deploy an Internet-wide layer-4 protocol other than TCP or UDP, and increasingly it will be harder and harder to deploy a truly universal L7 protocol but HTTP. That's my limited view of this issue, how things like IPv6 might affect it I have no idea.
SFTP/FTPS, SSH, SMTPS, etc.
* SFTP/FTPS (SSH)
* SSH (SSH)
* SMTPS (SSL, TLS)
You know that thing that gives security practitioners a bad reputation with engineers?
You're doing it.
As opposed to this, why not something more like "it's not uncommon for these two to be confused, but FTPS is FTP over TLS and isn't interchangeable with SFTP"?
I guess it is IETFs job to standardize stuff so that is what they do -- for a hammer everything is a nail, right?
P.S. I skimmed it, so perhaps it is worth a read, but nothing caught my eye really.
Not to take away from that, it's also very easy to get wrong as well.
- Not doing CA verification check[1]
- No alerts/monitoring of new certificates used by the IDP
- No verification on certificate-transparency of new certificates used by the IDP
- No online CRL checking
If you do not do these things, then your application-on-HTTP is insecure. Remember that it's very easy to get a new certificate if you can break IP either by MITMing the server or by BGP takeover of some space near the CA. Both of these things have happened.
If there's a way you can detect incorrect implementations, then blacklist them. For example, I use OAuth2 in my own systems, but I require clients retry (by randomly failing during onboarding) and implement fuzzing whereby the client_secret is invalidated if they accept an invalid certificate.
[1]: http://web.archive.org/web/20120317165131/http://forum.devel...
Specifically they are Google's Chrome (Android), Microsoft's IE/Edge (Windows), Apple's Safari (iOS/ macOS) leaving only Mozilla's Firefox (standing in for the Free Unixes). If there's a fifth it's Oracle, as custodians of Java.
A tremendous weight ends up resting on the shoulders of these vendors, all of them based on the US West Coast, and it's not as though this a power _seized_ by them, it's more something everybody else shirked.
> QUIC is an experimental transport layer network protocol [1]
You can run layer 4 on top of layer 4. (Just as you could say, run TCP layer 4 on top of HTTP layer 7.)
https://en.wikipedia.org/wiki/Mark_Nottingham
Perhaps this is meant as a public service, just to gather best practices. If you want to build your own thing, please go ahead, but don't call it HTTP.
I strongly disagree that ignoring these suggested BCPs disqualifies an application as "HTTP". As far as I'm concerned, if a typical middlebox will reliably pass messages in it, it's HTTP.
Is this really saying that, for example, I shouldn't have an API specification that says "to fetch a list of users", send a GET to `/api/v3/users`? What's the alternative?
I not particularly a fan, but it's fairly common. It was popularized as one of the pillars of the REST paradigm.
Paypal uses it for their API.
But yeah, it's kind of like functional programming: the ideas are super common and well understood. But by the numbers, the less pure imperative programming paradigm is overwhelmingly more commonplace.
[1]: http://openid.net/specs/openid-connect-discovery-1_0.html
This is also good to combine with requiring users retry and use multiple endpoints. This way you don't have to take risks with DNS and global routing failures potentially breaking your application.
If you use TLS to protect your application, I recommend further (randomly) generating a bad SSL certificate to make sure clients retry (and get a good one) so that you can detect anyone stupid enough to take advice like this[2]
[2]: http://web.archive.org/web/20120317165131/http://forum.devel...
How is that different than having only 1 version of the API and updating it? Clients code will still break if a structural data change is pushed (for example, removing a field from the responses)
> How is that different than having only 1 version of the API and updating it?
Using a service discovery document makes it easier to put /v1 and /v2 on separate servers/domains instead of putting both on all the servers/domains. Serving the service discovery document is cheaper than having your front-end server route all requests to backend systems (like using nginx or an Amazon ELB in application mode).
Putting applications on different domains is desirable if /v2 has performance gains (or cost gains) obtained by using the new protocol. Or if they're separate codebases.
> Clients code will still break if a structural data change is pushed (for example, removing a field from the responses)
It's very difficult to predict how a client will react if you haven't seen/vetted their implementation, but there are things we can do that go beyond service discovery.
For example:
My advertising service allows publishers (people who make web pages and content) to choose how to render my advertisers' ads in those pages. They do this via a server-side call -- I don't do any tracking of this, but I want to be sure that the client is doing the right thing because I'm still representing the brands.
I can look at the web page and sign off on it, but I want to make sure that it'll support a wide range of ads -- including ones I haven't thought of.
So I put a lot of energy into patterns like fuzzing the clients, and doing things that I hope will require they do things securely:
- I require retries so users always get an ad
- I randomly giving them a bad SSL certificate -- one that if they accept, I block their account. I usually only do this during onboarding or for my crawlers (recognised by IP) which check for sneaky stuff.
- I require certain utf8 characters to make sure the query is utf8 encoded
- I randomly insert random fields to make sure I can add new ones later
- I randomly reorder fields in the response to make sure they're not counting
I think that if you do all these things, and perhaps other things, and clients are used to it then you probably can have just one version of the API and update it. But this is hard work, and the enterprise is not (in my experience) very fond of hard work.
Things that HTTP makes more complicated:
1. It's a verbose protocol with more often than not a really bad ratio payload over metadata;
2. Client authentication is hard;
3. Server authentication is expensive;
4. It's bad at streaming;
5. It does not help a bit for RPC (no support for fan-out, quotas, fail-overs, retries, ...).
Building an infrastructure on HTTP makes it so unreliable that I came to consider one of the reason for Google technical superiority is that they early ditched HTTP completely out and used stubby for everything instead. Maybe they themselves view it as their secret sauce, which would explain why they've never published a comparative review (that I'm aware of)? Many people I've spoken to about this have expressed surprise that there is no HTTP flowing on Google's veins. To many, removing HTTP is like removing the solid ground; but to a former googler, loosing access to stubby and having to work with HTTP again is a painful experience that requires quite some time to accept.
Yes, when I worked there, it was interesting to realize that they take HTTP, and immediately convert it to protobuf.
By the way, server (and client, if needed) auth is implemented by the TLS layer. Do you propose using something else to secure other protocols?
Also: TLS is slow, and last time I asked for the best practice to do client auth on HTTPS the general consensus seamed to be "don't do it".
It is just a protocol for stateless server-client communication. Nothing more. What you build on top of it, is your choice. And you don't have to use it. But it is a mature protocol and it is well understood and it has a huge ecosystem of technology around it, that is just good enough for many use cases.
> 1. It's a verbose protocol with more often than not a really bad ratio payload over metadata;
Its a stateless protocol. You have to provide all the required meta data (Headers) in a conversation. Otherwise, it would not be able to be stateless. And yes, it does use text, because it would be extremely difficult to create a evolving protocol just by using some more memory-efficient binary tricks. Text is just better for comprehensible semantics. And that's even a good thing in pure machine-to-machine interaction.
There is also HTTP/2 which addresses efficiency and latency issues. But it trades that for simplicity of the transport method.
> 2. Client authentication is hard;
It depends. It can be as easy as sending an Authorization header. Or you can defer authentication to the transport layer (client certificates over TLS). Otherwise, I'd say: Best security is hard. And it won't get easier in other stateless protocols.
> 3. Server authentication is expensive;
That's just too generic. What does "expensive" mean? For what use case is it too expensive?
> 4. It's bad at streaming;
It depends what you mean with streaming. I think, you mean legacy binary streams? Yes, HTTP is not meant for that. But you can use HTTP for initiating your custom streaming use case. There is the "Upgrade" header. Or you just provide an URL to your streaming endpoint that uses something different than HTTP or even TCP, as long as the client can handle this stuff (e.g. RTMP).
Then there is something that is often misconceived as a utility for streaming: using chunked streams or using the Range header. Chunked streaming is just for fetching content of arbitrary size. And having a server support "Range" allows for fail-over / retries. Its not meant to support continuous streams of data.
> 5. It does not help a bit for RPC (no support for fan-out, quotas, fail-overs, retries, ...).
You can do RPC if you design your payload around it. See SOAP or even something like RMI over HTTP. Fan-out, quotas, fail-overs and retries. Everything can be done using HTTP. Does the specification itself solve those issues? No and it should not, because it would make it too complex and harm adaptation of HTTP.
Also you don't need to put all those headers in the requests/responses. And soon you will be able to override the horrible User-Agent one even in Chrome.
There are a whole lot of applications out there in widespread use that depend on UDP at the very least, eg. most multiplayer network games, just to use the simplest and most common example I can think of. Not everything can (or should) be stuffed into a HTTP transaction.
Game developers don't really have a problem with using UDP/TCP on random ports.
I want as much as possible to break for people who block those ports, and engage in other user-hostile actions.
Or I can create my own protocol which has none of those benefits besides possibly quick development.
Yeah, using TCP directly is great, isn't it?
The only times I know that non-HTTP works is video games and that is sorta hit-or-miss on one ISP I frequent, they block protocols they don't know for "security".
Making it HTTPS is an easy way to get around any potential firewall problems and have it work easily in corporate settings.
To use HTTPS, HTTP/2.0 or WebSockets is really not clever options when you can do secure, simple and more performant stuff in clear text over HTTP/1.1