The Future is Hypermedia APIs
emergentone.com
emergentone.com
Apart from parsing, I am the API client in this instance. I'm the one making decisions from understanding what services are about.
The universal API client seems to be a kind of silver-bullet-IA in many people's mind. I feel it's both impossible and not necessary to accomplish that. The complexity we have to handle to use APIs is already minimal.
What makes hypermedia interesting IMHO is (minor point) easier intra-linking within a webservice, and (major point) the ability within a service to point to resources from another one.
This will enable the distributed web we are all waiting for.
Your browser parses, yes, but then it goes and fetches images, css, javascript, iframes, etc. Following the analogy, these are 'api calls'. It knows to make these calls because the HTML defined the links, and for the HTML to be rendered the contents of the links are required.
I maintain the browser is the client - it responds to your actions, but it does so with a lot more activity than just parsing.
>What makes hypermedia interesting IMHO is (minor point) easier intra-linking within a webservice, and (major point) the ability within a service to point to resources from another one. > >This will enable the distributed web we are all waiting for.
Couldn't agree more. But without some standards, those links to other services may not be as useful as they could be.
This may well be the problem then, as we can see here and elsewhere. Many people like me don't understand. The claims are too vague. We need real-world exemples of how this can be applied with clear benefits. Mere transposition of the hypertext qualities to a medium intended for machine consumption will not do it for most of us.
It grabs the links from the content of the document it retrieved, using the rules of HTMLs structure to find them. That's both dynamic and discovered.
Speaking from practical experience using an API like this: it can result in a lot of extra round trips to get to your final destination. The indirection is nice purity-wise, but in practice do you really want to make 3-4 times the round trips to the server for every API interaction? Especially when you might be on a mobile network? All for the sake of flexibility you may never need?
I'd say supporting the discoverability for learning, and consistency etc is nice. But in practice concerns like this (and the DHH example, and others already in comments here) will crop up and clients will start building their own direct URLs. So the idea that you'll be able to change the URL structure without affecting clients is a pipe dream.
You can usually get around this. Many people make their resources have too much hierarchy; there's nothing inherently unRESTful about having a flat one.
You should _not_ be having 3-4 round trips for every API interaction. Each request is an interaction. Hypermedia APIs expose a workflow, not a data model.
I would say this is true of REST APIs. Not necessarily hypermedia APIs. An API that directly exposed a graph database as an HTTP-based API using hypertext/links within resources would still need to be classified as a hypermedia API, but would fail to adhere to the HATEOAS principle.
Edit: More specifically - a graph database that exposed itself with links for every edge and a resource for every node.
Secondly, when your data is hierarchical, it's nice for other reasons to reflect this in your URI structure. It's intuitive (and thus discoverable in its own way) and can make for human-friendly URLs (especially when your datatypes have natural identifiers).
It's pretty trivial to have the client hit the root on startup, and then cache that and never make another call. API roots don't change very much.
> First, are there then multiple roots?
Nope. Here's an example of what I call the 'hypermedia proxy pattern' in Sinatra: https://gist.github.com/3172911
I based this off of this talk by Jon Moore: https://vimeo.com/20781278 and demo'd it at the end of this presentation: http://oredev.org/2012/sessions/designing-hypermedia-apis
Basically, you can fold elements of the collection up into the parent, and the client will automatically make less requests. Jon's presentation goes from 14 requests for the first iteration to 2 on the first hit, 1 every hit thereafter, with no changes to the client.
> when your data is hierarchical, it's nice for other reasons to reflect this in your URI structure.
Sure! So provide both: one 'deep link' or full collection in the root (or wherever) response, but also serve the data as a separate resource that's hierarchical. Best of both worlds.
That's true and will work in most cases, but perhaps not when client sessions are short-lived.
> Basically, you can fold elements of the collection up into the parent, and the client will automatically make less requests.
Now it looks like my choices are to either fetch more data than I need, or make more requests than I need.
> Best of both worlds.
Well, both best and worst of both worlds. Each approach needs to stand up to cost-benefit analysis alone or I doubt it's worth maintaining both.
I do see advantages in these patterns, but I think some of them are more theoretical than practical, and I'll take the practical advantage I see today over the theoretical one I might need in the future.
Mostly, I think that would produce an ugly API.
The reason I've been leaning towards that solution is that it assigns the responsibility of locating any other resource, in one request, with full backwards compatibility, to exactly one location, freeing other locations to change in a more fluid and graceful way without the cruft.
Edit: Oh and because other URLs can have the context of their parent paths deliver some value or information. But then again URLs should be opaque =). Definitely worth thinking more about!
I don't see why the API would have to be ugly. The examples of "bookmark" link URLs given in the article are perhaps ugly (using query strings, etc.), but there's no reason why such URLs couldn't be "nice" ones. The article talks about things like "what if I want a URL structure like /User/projects/5 instead of /projects/5?", but that's a red herring: you could just as easily say "what if I want a bookmark query string that looks like ?user=User&project=5 instead of ?project=5?".
The bottom line is that as soon as you have a public API, you have identifiers that can't be easily changed because people need to be able to bookmark them; and those identifiers can't be too ugly because they are visible to users. So they should work OK as API URLs.
I feel like that is again happening here. I can not image how fully compliant REST api's would work in the real world. And when I ask for an example? Twitter, check the Twitter API! But no, that breaks the rules all over the place in the name of practicality. Oh yeah? You want a real example? How about the whole web? The whole web is your example!. That example is so far outside of instructive or helpful I don't even know where to start. I stand agape, unable to even speak. The way clients access my blog system should be based on the whole web? Even though it itself is on the web?
I'm done chasing prophets. In five years I do not believe this REST mania will exist anymore, and not because it is so ubiquitous as to go unstated. It will have died or changed drastically in the name of actually working. Bootstrap.rest. Either I am too dumb, or the movement is crazy, or both. But either way, I'm done trying to make things in a way I don't understand. The great and holy can thump their Fielding Bibles with faith that they will be taken home soon. Me, I've given up on salvation, and find the road to hell a much less exhausting one.
I don't think a single advocate of semantic CSS class names has changed their mind and now says there are no problems with using Bootstrap presentational CSS class names in your HTML. Just yesterday, on the front page of Hacker News, there was an article about the pain caused by overuse of Boostrap's unsemantic CSS class names: http://blog.pamelafox.org/2012/12/a-tale-of-two-bootstraps-l...
I don't know why you ever thought people were "screaming" about it, but if you're hearing about it less than about Bootstrap now, it's definitely from different people.
There are objective benefits for maintainability by designing your HTTP APIs to be RESTful, just as there are objective benefits for maintainability by choosing your CSS class names to be semantic, and they have been espoused by the creators of HTTP and CSS since their creation. It is neither "mania" nor "crazy".
Your child-like conversation with an unhelpful strawman notwithstanding, it is true that in the real world, software architecture always has to balance long-term maintainability against short-term ease of implementation. That doesn't change the fact that semantic CSS class names and RESTful APIs improve maintainability, always have, always will, people always have said they do, and people always will say they do.
Have you ever used GitHub's API?
$ curl https://api.github.com/But it's clear from GitHub's API that it will do nothing of the sort: those link relations (and even concepts) are very much GitHub specific, and no machine will ever be able to figure out what to do with the them, what methods or media types the endpoints support, and so on.
(I'm not sure that it adds very much over a text document that that gives the relations and how to generate endpoints, but at least it's cheap to produce, and not actively harmful.)
Consider a web browser, or an RSS reader. These are generic hypermedia clients that consumer standardized media types.
> But it's clear from GitHub's API that it will do nothing of the sort: those link relations (and even concepts) are very much GitHub specific, and no machine will ever be able to figure out what to do with the them, what methods or media types the endpoints support, and so on.
Right. We're not talking about AI, we're talking about generic re-usable components.
> (I'm not sure that it adds very much over a text document that that gives the relations and how to generate endpoints, but at least it's cheap to produce, and not actively harmful.)
The point is that by forcing you to define the communications protocol up front, you de-couple clients and servers. This means that they can evolve separately. Think about the RSS example: new versions of RSS clients can be made, and the servers don't need to be updated, and servers can update themselves and their responses and you don't need a new version of the client to handle it.
The other advantage is that if there are multiple services in the same product domain, and they use the same type, you get generic re-use of clients across services. Everyone spits out RSS, everyone consumes RSS.
With RSS, the client doesn't need to know very much--only one method is supported (GET), it corresponds to a click, and you'll almost always get one media type back (text/html). The IANA define some other generic link relations, but there seems to be a very big gap between generic link relations ("self", "back", and so forth), and application-specific link relations that are necessary for non-trivial apps.
How is GutHub's API useful to hypermedia libraries? A "library" that could be shared between GitHub and a different endpoint would be maybe 10 lines of code? HAL has a bit more structure, but we're talking maybe 100 lines of code. (Versus URI templates, which must be hundreds of lines, and with much more complicated rules.)
Well, RSS is just a simple example: its domain is small. Which is why it's nice to talk about. If you need bigger and more powerful things, than stuff gets complicated.
> With RSS, the client doesn't need to know very much--only one method is supported (GET),
I should have said ATOM and ATOMpub; it has a slightly more complex edit/delete workflow as well.
> The IANA define some other generic link relations, but there seems to be a very big gap between generic link relations ("self", "back", and so forth), and application-specific link relations that are necessary for non-trivial apps.
Right. The idea is that you use app specific ones at first, then, as you find they're useful and stable, you make them a generic one.
> How is GutHub's API useful to hypermedia libraries?
This is backwards. It's more like "how are hypermedia libraries useful for implementing the GitHub API."
Ok, but do you think there will will ever be a generic definition of "user" or "product" or "photo" that can access the full experience of e.g. Amazon, Google, Facebook, Flickr and GitHub?
Regarding Atom, Google adopted a extended version of Atom in for some (but not all) of their products, but no-one else adopted it in almost a decade, and Google are now deprecating it. (I'm sort of blurring the difference between link relations and media types, but they seem entwined.)
URIs and shared media types are great for interoperability, but my guess (we'll see if it comes true) is that pretty much anything beyond that isn't going to get traction. Hypertext was around before the web, but I think a big reason it took off is because had very loose interoperability demands. (e.g. links can break.)
The closest I can think of is the way some APIs handle pagination (GitHub and Recurly for example) by using Link: rel=next/prev/start headers rather than through GET parameters, which in principle would allow a generic REST client to iterate through results on its own.
curl https://api.balancedpayments.com/v1/marketplaces/TEST-MP6IEymJ6ynwnSoqJQnUTacN -u 7b7a51ccb10c11e19c0a026ba7e239a9:
You can see in our python client how we then parse - https://github.com/balanced/balanced-python/blob/master/bala... - the result of this which then consumes these URIs and turns them into dynamic resources attached to whatever object you're looking at.
This means you can do this:
debit = balanced.Debit.find(uri)
debit.account.cards.all() # get a list of all cards associated with the account that created this debit
We make changes to our internal services all the time and HATEOAS allows us to ensure our services are properly functioning without breaking contractual API obligations.
I can see why most developers resist the urge to try hypermedia APIs and it's honestly because the tooling to build services and clients just aren't there.
This is why I'm a big fan of the https://github.com/rails-api/rails-api project. It's essentially accepting this deficiency, has some brilliant minds behind it and as a consequence, I'm sure the tooling to be built around it will be amazing; it's definitely one to follow.
Some more resources:
Please see my comment here: http://news.ycombinator.com/item?id=4949311
Also, Balanced (YC W11): https://www.balancedpayments.com/docs/overview#storing-the-u...
We're experimenting with more :)
I really need to buy you a beer sometime.
* Mike Kelly (the author of the HAL spec) is not aiming for a universal API client http://news.ycombinator.com/item?id=4949357 (and also seems to be under the impression that no-one else is either).
* A universal client seems impossible because APIs will typically need to use service-specific link relations for all "interesting" relations (e.g. product, author). You might be able to share generic link relations (like those in IANA's table: http://www.iana.org/assignments/link-relations/link-relation...), but sharing only these doesn't seem to be worth the effort.
* Hypermedia-powered experimentation is kinda cool, but you're still going to need something to drive it (see e.g. Kelly's HAL browser http://haltalk.herokuapp.com/explorer/hal_browser.html#/), at which point you might as well write a full-fledged API dashboard.
HTML is a hypermedia media type. A server spits it out, the client (a browser) consumes it.
A browser is a generic hypermedia API client, for services that expose HTML.
> it's not an API in the sense most people understand
Neither are hypermedia APIs in general.
How can I point the HAL browser at foxycart's API? Or even see Foxycart's browser? I can only find standard hand-written documentation.
In theory, the URN solution could work, but it's far more complex and requires more maintenance than simply having URLs that can be relied upon. His solution to the problem of breaking bookmarks is also a lot of work for consumers of his API--they have to two codepaths: one for when the URL works like it should, and another for when it changes and they have to use hypermedia to find it again. Again, most consumers of the API aren't going to give a shit. They are going to hardcode URLs and complain when you break them.
Not only that, but consider the implications of an API that requires you to start from the beginning each time for mobile traffic. Each request over a mobile connection takes an eternity due to high latency. You're seriously telling me that you want to force users of your API to make N * depth of resource requests, instead of just going straight there, when each request takes 100 ms or more? Okay, so maybe you can cache it, but that just means the user's first impression of the app is that it takes 500 ms just to do the first thing. Pretty shitty first impression.
The comparison to a browser doesn't hold much water for me either. Users (and search engines) will save your URLs, so if you break them, you will lose traffic. If I try to go to my bookmark and get a 404, you really think I'm going to take the time to find your page again? Maybe, but it's much more likely I'm going to shake my head and close the tab. Of course Google will eventually find your page again, but it will have lost the SEO points it gained, which means you'll have to start rebuilding your SEO from scratch.
The other issue with the browse as hypermedia engine that he leaves out is the human brain. The human brain is excellent at extracting information from text data, so there doesn't need to be any kind of common standard or format to browse a webpage. The brain just figures it out that when you say click: [here], [here] is a link that goes to whatever you were just talking about. That dramatically reduces the burden on webpage authors to conform to any kind format. His theoretical (well, implementations certainly exist) universal REST client only works when every API you want to consume standardizes to presenting the information in the same format.
I drank the HATEOAS Kool-Aid when I first learned about REST too, and then I built a beautiful API that used it. What did the developers ask for? A list of URLs.
The future is almost never $elegantly_designed_complex_system. It's almost always a pile of garbage, with a few humans sitting around sorting through the garbage to find the thing they want.
A few years ago, before Rails made "REST" popular, this exact same statement was made. "Nobody is going to want to learn about PUT and DELETE. They just want to do everything over POST."
> They will hardcode that URL in their app, too,
This is an education problem. We're still in the early days of this stuff.
> One for when the URL works like it should, and another for when it changes and they have to use hypermedia to find it again.
This is a tooling problem.
> You're seriously telling me that you want to force users of your API to make N * depth of resource requests, instead of just going straight there, when each request takes 100 ms or more?
One thing that's nice about hypermedia is that you can lead clients to wherever you want them to go. You're free to change things at any time. Have a mobile app? Make less features, and make the request paths shorter. Have a desktop app? Make them longer, add more stuff. It's up to you. Nobody says that you have to complete N steps to do anything unless your server does.
> Users (and search engines) will save your URLs, so if you break them, you will lose traffic.
Right. So what do we do on the web? We 301 redirect until we don't have enough traffic to care anymore. Same thing with your API.
> The future is almost never $elegantly_designed_complex_system. It's almost always a pile of garbage, with a few humans sitting around sorting through the garbage to find the thing they want.
Luckily, Hypermedia is the _opposite_ of Big Design Up Front. It lets you evolve over time. Partial application still yields benefits.
Please see my comment over here: http://news.ycombinator.com/item?id=4949311 GitHub, BalancedPayments(YC W11), and others are finding real benefits today.
> A few years ago, before Rails made "REST" popular, this exact same statement was made. "Nobody is going to want to learn about PUT and DELETE. They just want to do everything over POST."
> This is an education problem. We're still in the early days of this stuff.
The developers I was working with also had a lot of trouble with REST. I had to explain repeatedly what PUT, DELETE, GET and POST were (they weren't really familiar with HTTP at all), and that they weren't doing RPC calls. On the one hand, I feel REST is a simple and elegant concept and its not much to ask someone to understand, but on the other hand, when I build an API, I want it to be easy to use and accessible to developers of wide skillsets and levels of experience. To me, sacrificing some of my desire for elegance is worth it if it reduces the level of “education” required.
> One thing that's nice about hypermedia is that you can lead clients to wherever you want them to go. You're free to change things at any time. Have a mobile app? Make less features, and make the request paths shorter. Have a desktop app? Make them longer, add more stuff. It's up to you. Nobody says that you have to complete N steps to do anything unless your server does.
Either I'm not understanding you, or this seems totally backwards to the way APIs should work. You build an API so that users can do things with their data/your service that you haven't thought about, or cover edge cases that aren't interesting to you. I feel like my API should enable people to build the things they want, not limit and encumber them so that I can have my pretty hypermedia. In other words, I'm not interested in leading clients places. I'm interested in where they lead me.
The best hypermedia API that you know of is ATOM and ATOMpub. ATOM is data exchange, but it also describes how to edit entries and delete them. This allows many services to expose ATOM, many to consume it, and everything is peachy keen.
The issue with 'just exposing data' is that you couple the business processes that your APIs do to organizations outside of your control. By exposing your processes (the 'state machine' of REST), you remove that coupling.
You may want to see this talk: http://oredev.org/2012/sessions/designing-hypermedia-apis
Of course, there are many APIs where 'just exposing data' makes sense. Hypermedia won't help you there, so don't use it. But if you want to build lasting, scalable, evolveable systems, then hypermedia is for you. If you just want to expose some data, then it's not.
You should try using the SQL analogy instead of RPC, as this is much closer to what a "traditional" developer might expect. With REST, one is not doing a procedure call, but one is querying to create/retrieve/update/delete data. Of course, there might be some procedures behind it, but to a client it's just a data query (just like SQL has stored procedures of its own).
Maybe you should've built them a client too? https://www.braintreepayments.com/braintrust/when-rest-isnt-...
I've heard great things from people updating their API clients to use our new hypermedia stuff. So we're pushing forward with our hypermedia experiments at GitHub. People are still free to optimize for speed (reduce the number of HTTP calls in environments where it really matters) or even development speed (hardcoding a few API calls instead of dropping in a full client library) if they like.
I'm not strictly advocating HATEOS. Hypermedia and HATEOS are related but not the same concept. I am very strongly advocating hypermedia, as I believe it it greatly reduces the complexity of interacting with an API.
Never having to build a URL by hand is great! Being able to just grab a field from a resource, and go do a GET on its value is very easy to code, even if that process involves finding a link with a certain relation. You speak of complexity, and I'm speaking of convenience.
With that convenience also comes the ability to automatically crawl the data with good solid knowledge of how each resource relates to others, enabling the ability to create massive graphs of information. Like wikipedia has for humans to read, but structured and standard for computers to parse.
> His theoretical (well, implementations certainly exist) universal REST client only works when every API you want to consume standardizes to presenting the information in the same format.
That sounds like a great day to me.
This kind of thing sounds great for exploratory development, but undesirable (for the reasons I outlined previously) for an actual production app.
> That sounds like a great day to me.
I hope you're used to waiting :).
We actually do this in our production app. The only URL we actually hardcode in the app is the root URL for the account currently logged into. (And a few other urls unrelated to the account, but that's honestly just pure laziness).
User logs in - we get the account (root resource)
Bootstrap the JS - we get a subresource of the account and request it, populating the initial UI
User interacts with UI - generates a GET to a URL we got from another resource, repopulates the UI with relevant data from that resource. Rince, repeat.
Doing this has made changing our API structure over the course of development a lot easier.
> I hope you're used to waiting :).
I am :). But I look at how much faster html5 went from concept to production because it was pushed by the browsers, and I experience an $optimism++. Things are moving in the right direction, IMO.
For example production apps, see this comment, and the one it links to: http://news.ycombinator.com/item?id=4950877
(Ohhh, hypermedia. ;) )
If the answer to this is "educate people" then that's the wrong answer. Or maybe it's the right answer if you're an IT type that loathes their users. I know I might sound crusty about all of this but how many years are we going to chase an idea that only a few people seem to understand?
Can we just admit it, right here, right now: all of this is academic. We should be focused on extracting patterns and practices that provide value and cycling on that.
The rest is noise.
Basically, I'm thinking that the ultimate benefit to having somewhat standardized interactions with our web APIs (with more included metadata) is that they can be mined, discovered, and manipulated intelligently by algorithms.
It helps people, sure, but just like using contact microformats helps give the user a phone number they can click to call, it's equally if not more important that it gives a crawler an easy way to say "hey look, a phone number!"
This is the main criticism I have with DHHs comments on hypermedia: they seem to focus too much on the human consumer (and often the programmer) and not enough on the algorithmic potential.