RESTful API Server - Doing it right
blog.mugunthkumar.com
Tips on how to get your API server done right, especially when you are a startup.
blog.mugunthkumar.com
- Hypermedia as the Engine of Application State (HATEOAS)
- Returning concrete mediatypes (standard is great, custom is OK) instead of generic 'application/json' or 'application/xml'
It also contains some misunderstandings:
- Sending a sha1 of the password to the server is bad, for two reasons:
- - First, and more importantly, if you send the hash of the password over the wire, the hash becomes a plain-text password! Seems strange? Think about it: if the attacker somehow gets the hash, he can just use it as-is to authenticate to your API - he doesn't need to crack it.
If you're using HTTPS, just use Basic Auth. Otherwise, you should implement (read: use a library!) proper request signing.
- - Second, sha1 by itself is pitiful - it can be cracked in mere seconds. Use bcrypt (this, on the server).
Once you know the top level objects and actions in your product,
designing the endpoints becomes easier and clearer. For example,
to “add” a new venue, you would probably have to call a method
similar to /venues/add
In REST, there should be a single endpoint. Everything else is done by navigating.
And in REST, there are no method calls! As Roy Fielding would say, this screams RPC! To add a venue you should just POST to /venues/ (URL which you discovered by navigating the API).
Read his blog post with some of the rules that make a REST API: http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hyperte...
this isn't REST and only serves to confuse things further
I don't fully understand what it is about HATEOAS that people find difficult or unconvincing. They must use web sites every day. When you want to buy a book at Amazon, do you read their documentation and then construct a URL? No, you go to amazon.com and click on shit.
It's clearly non-obvious to work with an API primarily as a set of documents rather than as a set of "methods" to be called - I have no idea why it's non-obvious at the start as it is exactly how the Web works and it's pretty obvious and elegant in retrospect.
If you're not using your own API, then it's probably broken.
As a browser user, I can generally look at a web page and figure out what's a clickable link. The HATEOAS constraint is rather trickier than that: it requires that a client understand what is a "clickable" link in an HTTP response based on nothing other than the content type and the response body.
For the past couple of years I've been building RESTish web services using JSON. They use HTTP methods and status codes appropriately, each resource has a URL, my response objects include URLs to subsidiary resources, and so on. However, the services I've built are not RESTful because JSON is not a hypermedia content type.
To be truly RESTful, I would need to adopt and/or define a hypermedia content-type using JSON and make sure my response objects conform to that content-type.
You just say "This is the format application/vnd.myservice.userprofile+json". In it there's an object, which contains the key "avatar_url", which has the URL of the user's avatar image.
Frankly, I don't see what's tricky about this.
I'm not writing this to dump on REST. I still think it's the right approach to build a web service over HTTP, but I'm also conscious of the diminishing returns on each incremental step toward pure RESTfulness. I'm just trying to make a web service, and suddenly I'm defining new content-types?
For a client, is it worth the trouble to learn a new content-type to determine what's a hyperlink in my JSON response, when they can just look at the response and see something like the following?
GET /articles HTTP/1.1
{
"articles": [
{
"url": "/articles/1",
"title": "This is my first article",
"pubDate": "2012-01-05T08:39:47.625000
},
{
"url": "/articles/2",
"title": "This is my second article",
"pubDate": "2012-01-13T11:07:35.219000
}
]
}
If the server generates custom mimetypes, clients have to ensure they "Accept:" the correct mimetype for every request. And if you have multiple data types (and you will have multiple data types), then this "Accept:" header will vary for every request. Getting this right is an irritant to client developers with no actual benefit to them, and a great potential source of bugs.
Much easier is if the client can just send "Accept: application/json" or "Accept: application/xml" for every request.
If the server generates custom mimetypes, and you update your API to v2 which produces an article listing in an incompatible format, you either have to introduce a version number into your mimetype (which seems messy and pointless busywork), or stick with the same mimetype and accept that your custom mimetypes aren't actually particularly meaningful as their meaning can change substantially over time (so why did you have them in the first place?)
Much easier is if the server had just used "Content-Type: application/json" or "Content-Type: application/xml" which will always be correct.
No, he doesn't. First, the server can send whatever it wants - nothing breaks if you reply with your custom media type to a generic "application/json" request.
Second, even if you want to be correct, that's what wildcards are for: the client can simply send always the same header:
Accept: application/*+json
I still don't get what's the problem with changing a string. I think you're using a bad framework.
This is a good point, though, not sure why it escaped me:
Accept: application/*+json
Probably. This isn't why, though.
One real advantage I can see is to use the media type for versioning, but even that doesn't remove the need for a client to intervene if they want to upgrade their code to work with the newer API version.
So it's not really clear to me that a custom media type is significantly more useful for the client, even if it is more 'proper'. When a constraint feels ceremonial, I become suspicious that it exists to drive purity-for-its-own-sake rather than a real benefit.
For a client, is it worth the trouble to learn a new content-type to determine what's a hyperlink in my JSON response, when they can just look at the response and see something like the following?
Personally, I think that mindset will hold us back. People still think of RESTful services as closed silos instead of open, truly uniform interfaces, and that will prevent us from growing, in my opinion.
Your view is of a developer painstakingly coding against each API (s)he wants to use, manually prodding and poking it and writing a custom library that doesn't work with any other service.
My view is of generic, easily adaptable clients, using machine-readable specification of formats¹ and providing graphical UIs where even non-programmers can input the single endpoints of various services and easily combine the data and functionality to create applications that cater to their specific needs.
I see a service like IFTTT[1], but more flexible and where the services are not hardcoded but imported dynamically by the user and then strung together to fit their needs.
Using mediatypes, even if custom, is a step towards that integration and standardization that makes this possible.
¹ RDF seems like the best bet to me - possibly encoded as JSON
[1]: http://ifttt.com/
In general, REST principles give more than they take, which is why I try to follow them; but you're always going to have to find the sweet spot of abstraction that provides the right amount of power and flexibility combined with the right amount of discoverability and predictability.
> What's the cost of just changing the mediatype to some different string?
I'm happy to change a server's content-type from application/json to application/vnd.my-adhoc-service+json, but that won't change the client's need to read my documentation and play with the service before they can figure out how to use it. That's true even if I move the documentation out to an RFC.
One real advantage I can see is to use the media type for versioning, but even that doesn't remove the need for a client to intervene if they want to upgrade their code to work with the newer API version.
So it's not really clear to me that a custom media type is significantly more useful for the client, even if it is more 'proper'. When a constraint feels ceremonial, I become suspicious that it exists to drive purity-for-its-own-sake rather than a real benefit.
Is RSS difficult and complicated to learn and use? Now think what a pain it would be to develop a feed reader if each blog had its own "simpler" ad-hoc schema.
Standards don't have to be complicated, especially if they use an extensible encoding like JSON or XML so that instead of having the kitchen sink they can remain simple and be extended for specific purposes if really needed, without affecting generic clients.
I'm happy to change a server's content-type from application/json to application/vnd.my-adhoc-service+json, but that won't change the client's need to read my documentation and play with the service before they can figure out how to use it. That's true even if I move the documentation out to an RFC.
The end goal is for different services to agree on standard formats for common resources. Using custom mimetypes is a step in the right direction because it decouples the format recognition and parsing from the service-specific code - it allows clients to use more modular code that works across different services with plenty of reusable code.
Is it terribly important? No. But since the costs are essentially none - changing a couple of strings here and there - not doing it seems unreasonable to me. You asked why, I ask why not?
> Is RSS difficult and complicated to learn and use?
No, but RSS is a schema with a specific, narrowly defined purpose: to syndicate article-based web pages (and even so, there are 5 or 6 different versions, not including Atom - though your point stands and I don't want to get sidetracked over a quibble).
A more apt comparator may be WSDL, which theoretically allows a SOAP web service's remote procedure calls to be discovered programmatically but in practice is such a messy, complicated, leaky abstraction that it's often easier just to build XML templates manually and POST them to the endpoint.
I'd be happy to use and support some kind of lightweight schema for JSON that puts URLs in predictable places in the response object and still allows flexibility to provide the functionality a given service needs. I'm not aware of anything that fits this description, but I'm certainly open to it if you have any recommendations.
> You asked why, I ask why not?
The model of HTTP resources and methods is useful: it unambiguously delivers real, tangible benefits to the developer and the client over ad-hoc protocols riding on top of HTTP like `GET /create_user.php`.
Similarly, it makes sense to use the HTTP status codes we already have instead of reinventing your own arbitrary set of status codes and, e.g. sticking them somewhere in the response body. If someone e.g. does a POST request on /users/1337 to create a user, it makes sense to return a 405 status. (However, I tend to put status codes in the response headers and the response body, just to make it easier for clients to notice what's going on.)
It also makes sense to put URLs inside the response body in an obvious way so that the client can "click through" to subsidiary resources in a manner analogous to how web pages do it. Again, it makes the web service more or less self-documenting to the client, especially if it uses URLs and HTTP Methods in a sane, predictable way.
I'm struggling to understand the benefit of a custom media type beyond merely satisfying one of Roy Fielding's constraints. If the only reason to use a custom media type is to satisfy a constraint, the constraint might be more ceremonial than useful.
If two web services are identical except one serves `Content-Type: application/vnd.some-arbitrary-format+json` and the other serves `Content-Type: application/json`, I get suspicious about the benefits of REST when people say the latter is doing it wrong and doesn't deserve to be called RESTful.
A more apt comparator may be WSDL, which theoretically allows a SOAP web service's remote procedure calls to be discovered programmatically but in practice is such a messy, complicated, leaky abstraction that it's often easier just to build XML templates manually and POST them to the endpoint.
Well, two points:
Firstly, I think WSDL (and WADL) are working at the wrong abstraction level. I think one of the reasons why REST is cleaner is because unlike SOAP it's data oriented, not RPC. And I believe it's possible, as long as your API is data oriented too, to map that to generic formats that don't feel messy.
I had already mentioned RDF (Resource Description Framework), but I'll mention it again; RDF is a layer between the serialization and the concrete schemas, giving very specific but open ended semantics to its documents.
Unlike WSDL, it's very simple: it consists only of four concepts: triples, which are composed of objects (which is always a resource, and is represented as an URL), subjects (which can be other resource or a literal value) and a predicate (which relates that object with the subject in some way, and which are specified by a schema). Then, a document is just a bunch of statements about one or more resources.
This model is hypermedia (since it clearly points to resources), and it's at the same time constrained enough to not end up being messy, and open enough to define mostly anything. I think it really captures the sweet spot.
There's no JSON serialization, unfortunately, but I like Turtle[1].
---------
The second point is that even if this day dream of mine never comes to pass, I think we'd still benefit a lot from using more of those standard "schemas with specific, narrowly defined purposes" and keep the custom formats to when they're really needed. Instead of defining your own user profile schema - which 99% of the APIs out there need - why not just share a single format, which can be easily:
1) Parsed by generic clients, libraries and plugins
2) Shared between services and applications
But for that, mediatypes make perfect sense, since they make it easy for a single HTTP client to dispatch the responses to the right readers, without involving a bunch of hairy rules that say "when response is JSON and the service being called is X, call parser Y".
This post is against WADL, but I think it expresses that power of using proper mediatypes: http://bitworking.org/news/193/Do-we-need-WADL
I'm struggling to understand the benefit of a custom media type beyond merely satisfying one of Roy Fielding's constraints. If the only reason to use a custom media type is to satisfy a constraint, the constraint might be more ceremonial than useful.
I might be wrong, but I'm convinced some great things could come up from it. Unfortunately, I doubt that anyone will seriously explore the possibilities unless there are already many APIs with support for it, and in typical Prisoner's dilemma style, very few API developers will consider supporting it unless they see a short term benefit.
If two web services are identical except one serves `Content-Type: application/vnd.some-arbitrary-format+json` and the other serves `Content-Type: application/json`, I get suspicious about the benefits of REST when people say the latter is doing it wrong and doesn't deserve to be called RESTful.
Well, I think as long as it supports HATEOAS it's RESTful enough, but on the other hand, I can see the their point. There's nothing wrong with not fully implementing REST, but still calling it REST gives false expectations to the clients.
Of course, at this stage nobody expects a RESTful API to be actually RESTful, but then again that's the problem, isn't it?
Well, sorry for the WOT :|
// Ahem.