API Design Guide | Hacker News Reader

API Design Guide

cloud.google.com

824 pointsandybons183 comments

I would like to add Microsoft's API Guidelines [1] here, which is also a well written document and can be helpful to anyone designing an API.

[1]: https://github.com/Microsoft/api-guidelines/blob/master/Guid...

It's interesting that both of these guidelines kind of reject HATEOAS by mandating explicit versioning. It seems that HATEOAS was never really a thing. It's just too complicated to implement in practice. In that sense, REST in practice has always been just RPC without a clear spec for procedure call like XML or JSON RPC.

I think, there is just too much bias involved. At least thats what I am experiencing. Not using HATEOAS ever, but complaining about it makes me angry.

My preaching: If you can write a client by using a semantic document format (e.g. HTML, XML+XSD, Json-Ld), you end up with a more elegant and stable implementation. And as a provider of such an API, I spend more focus on the surfacing domain than the structure of my resources.

It makes me sad, that even Google does not try.

I wrote about API design (with a similar rejection of HATEOAS): http://www.vinaysahni.com/best-practices-for-a-pragmatic-res...

In short: humans can follow links, even as a website goes through significant changes. Code can follow links, but can't make clear independent decisions when significant changes happen to the API.

[Updated for clarity]

Code can follow links as well, as long as the semantics doesn't change.

Aside from versioning through mimetypes, which I believe is a really bad idea, I find HATEOAS to be a beautiful concept, although not very useful in practice.

It's a good place to start though. Trying to design for that can help you shape your API properly, just like SOLID or TDD can do for code.

Why do you think versioning through mimetypes is a bad idea?

I should probably have moderated that statement, but mostly I think it's bad because it makes the versioning aspect inaccessible from the lowest common denominator, the browser address bar.

Versioning through the url and versioning through mimetypes requires the same amount of work from the user, but the former is a lot simpler. If you then also take into account, that many http clients have poor support for manipulating headers (SAP and PowerShell (earlier version) for instance), then choosing mimetype versioning makes it harder (impossible) for your users.

I also prefer putting the version in URL. I find it to be practical and easy way to version your API. I've also been told that it's not the right way to do it.

There's no "right" or "wrong" way tp version APIs. There's only peoples opinions. I'd suggest that if someone is saying you're doing it wrong by putting it in the url, you should probably learn to ignore them as they can't tell the difference between their own preference and facts.

> mostly I think it's bad because it makes the versioning aspect inaccessible from the lowest common denominator, the browser address bar.

That also rules out using any HTTP method other than GET. The browser address bar doesn't seem relevant here, and if it were, then it would rule out the vast majority of APIs.

Versioning through the URL breaks interoperability of clients using different protocol versions. If a client using v1 communicates with a client using v2, then they will see two separate sets of resources and they would never have the same identifiers for (what should be) the same resources.

I suspect that we are designing for different clientele.

I'm currently in charge of a pretty classic, GET things out, and POST things in API. If my API supports that some non-IT office guy can get his daily report through the browser, and the API at the same time has the power and flexibility needed for someone with IT skills to do more then I've won. Both me and my users gain by me taking extra steps to make it as accessible as possible.

I also fail to see how versioning through urls break anything, except if the newer client no longer has support for v1. In both versioning schemes you specify which version of a resource you're sending/wants to receive, in one scheme you do it through a header, in the other through the url.

>> If a client using v1 communicates with a client using v2, then they will see two separate sets of resources and they would never have the same identifiers for (what should be) the same resources.

Maybe we're defining versioning differently. I strive to make resource types backwards compatible, but if I can't then it's either a new version or a creating a new more specialized resource type for the specific problem.

> If my API supports that some non-IT office guy can get his daily report through the browser

Most APIs I work with wouldn't work this way even for read-only access because of authentication requirements. I guess you're using something like HTTP digest auth with usernames and passwords?

Even so, that still works for your use case as long as your users are happy with a default version. e.g. requests without a specific version get the latest version, or requests without a specific version get version 1, or requests without a specific version get the latest stable version that changes from time to time. I don't think I've ever come across non-IT office guys that have more complex versioning requirements than that.

> I also fail to see how versioning through urls break anything, except if the newer client no longer has support for v1.

Here's an example:

Client A (using version 2), talking to Client B (using version 1):

> Here is a friend suggestion for person https://example.com/v2/people/foo

Client B:

> Okay, I'll add them to my contacts. My contact list is now:

   https://example.com/v1/people/foo  
   https://example.com/v2/people/foo (error: cannot parse)

Had the URIs truly been identifiers, this would have looked like:

Client A (using version 2), talking to Client B (using version 1):

> Here is a friend suggestion for person https://example.com/people/foo

Client B:

> Hey, I already know that guy!

Once you break the identity part of URIs, you start getting a tonne of these awkward problems to work around. URIs are meant to be the primary keys of the web, and you're going against the grain of the medium when you break that.

To put it another way, imagine if websites had to change every page URI from /html4/ to /html5/ when they switched versions. That's a whole lot of work and breaking things for no reason. I'd rather follow how the web was designed to work and not create so much extra work.

>> I guess you're using something like HTTP digest auth with usernames and passwords?

Basic auth (over https only) actually, I'm even advocating it. We support expirable tokens as well.

>> requests without a specific version get the latest version, or requests without a specific version get version 1, or requests without a specific version get the latest stable version that changes from time to time

The only way for me to go would be option 2: requests without a specific version get version 1. Any other choice and I'd have customers API integrations breaking, because many of them would not have specified their version mimetypes correctly and would therefore get the default version, which would at some point diverge from their expectations.

>> URIs are meant to be the primary keys of the web, and you're going against the grain of the medium when you break that.

In that context your example makes sense, I just don't see that as realistic problem. In real life URI aren't as static as we pretend. How many companies would make sure the resource urls are still valid after changing the company and domain name?

The real identifier in your case is "foo" and "people" is the type. Accepting that sets you free to do versioning as you please and to save 90% characters in all ids :)

In the end we should design apis for our users, not for our selves. My clients are always non-IT departments, often in enterprisy environments, and if they can get 2-4 hours allocated from IT in the next 6 months to do some API integration, they are lucky. So I design my api for maximum simplicity, and maximum longevity of the using scripts, because if I wanted to make a breaking change, I'd have to wait 6 months before all customers could be expected to have transitioned.

Most of my api users couldn't tell you what a mimetype is btw., but a v2 in the url would make complete sense.

I get where you're coming from, on our latest api version I started out wanting to do full HATEOAS (Bought and read this great book http://www.designinghypermediaapis.com/), but in the end I had too much trouble getting my test customers migrated, so I had to scale back on the desgin goals. Scaling back to what worked for everybody was very informative and the api is now, imo, the better for it.

If I need to bump the vesion it'll be with a vx in the url. An id is a simple non-globally unique thing, that you use to build a url etc.

> Any other choice and I'd have customers API integrations breaking, because many of them would not have specified their version mimetypes correctly and would therefore get the default version, which would at some point diverge from their expectations.

"Some other people might write code that does things incorrectly and then their software might break" is something you can say about any API style, it's nothing special to media type versioning. If many of your customers are writing code that integrates with your API incorrectly, you don't have an API design problem, you have a poor communication problem, and that's going to hurt you in all kinds of ways.

> In real life URI aren't as static as we pretend.

They are static if you don't deliberately break them; I'm just pointing out some of the consequences if you choose to.

> How many companies would make sure the resource urls are still valid after changing the company and domain name?

Companies are laser focused on this if they rebrand. Companies love their search rankings. But how common is it to rebrand? Optimise for the common case, not the uncommon ones.

> The real identifier in your case is "foo" and "people" is the type.

You're turning URIs from an opaque identifier clients can use as-is to a compound structure that clients have to parse, manipulate, and (re)generate. You're putting more logic on the client, and thus giving client developers more rope to hang themselves with immediately after saying they can't be trusted to get things right.

This is not designing for maximum simplicity. Designing for maximum simplicity is to have an API that says "here's a link; follow it", not an API that says "here's an ID, generate a URI from it according to these rules we set out in our documentation, that are different for every type of resource". The latter is more work and more error prone.

I hope it's clear from my previous comments that I don't actually disagree with you on most things, I've just had to make compromises that makes things more accessible to my clientele.

>> "Some other people might write code that does things incorrectly and then their software might break

Welcome to my world :)

>> is something you can say about any API style, it's nothing special to media type versioning

True, but mimetype versioning still makes things more complicated for my average user.

>> If many of your customers are writing code that integrates with your API incorrectly, you don't have an API design problem, you have a poor communication problem, and that's going to hurt you in all kinds of ways.

Sorry, but no. We have full Swagger interface, examples for all major operations in 4 different languages, many pages of documentation and tutorials, and very clear and informative error messages for most situations. Still people will write and say it doesn't work, without having read any docs or even looked at the response.

>> This is not designing for maximum simplicity. Designing for maximum simplicity is to have an API that says "here's a link; follow it", not an API that says "here's an ID, generate a URI from it according to these rules we set out in our documentation, that are different for every type of resource". The latter is more work and more error prone.

I'll reiterate, we're probably designing for different clientele. I'd wager my old Amstrad that most of my users would strip any url'y parts of ids I'd return and just store, what they'd perceive to be, the "true" id in their database, because they only have 20 cloumns allocated in their weird mainframe system, named info1-10 and extra1-10, and the latter only allows 32 chars.

But also consider this: A user uploads a resource through the perfect HATEOAS api, he get's back a big object with a unique url id, and links to all the actions he can perform. It doesn't really makes sense for him to store anything other than the URI though, because I might add more actions later, so he'd have to requery the URI to get links to the latest actions anyway. So for any operation he wants to perform, he should GET URI, parse and follow link with appropriate parameters. Also, in that scenario my system gets hit twice for any operation. Compare that to: Replace parameters into https://example.com/people/{id}/poke and fire request. I'd argue that the latter is conceptually simple.

Very soon we'll start on a new api for our ui, and on that there will be no compromise :)

When you say client do you mean how humans passing around URLs? I can't come up with any reason why two client applications would be sharing full API URLs directly with each other, that seems like very poor design. Even under some sort of weird direct connection situation where the server wouldn't be generating the paths using the URLs as GUIDs seems like a really strange decision.

As for auth a simple token as Github does is enough, and github lets you use it as a header or a get param.

Your choice of HTML 4/HTML 5 for comparison seems aptly poor. Instead of clear versioning, old browsers get HTML they don't understand and fail unexpected, unpredictable and ungratefully ways. The exact opposite of what we should want for API.

> When you say client do you mean how humans passing around URLs?

No, I mean API clients.

> I can't come up with any reason why two client applications would be sharing full API URLs directly with each other, that seems like very poor design.

URIs are the primary key for resources on the web. If two clients want to talk about the same resources, then they need to agree on an identifier, and that's exactly what URIs are designed for.

> As for auth a simple token as Github does is enough

The design criteria in this example is "Can be typed into a browser address bar by a non-technical person". I don't think auth tokens meet that criteria.

> Your choice of HTML 4/HTML 5 for comparison seems aptly poor. Instead of clear versioning, old browsers get HTML they don't understand and fail unexpected, unpredictable and ungratefully ways.

You're conflating two different things there; how versions are communicated, and the compatibility strategy. Whether or not HTML did a good job of maintaining compatibility between different versions, do you agree that changing the URI of every web page on the web to communicate when they change HTML versions is a bad approach?

Thank you for your write-up on API design. It's well-written, concise and to the point. I spent quite some time reading it while learning best practices about REST API design last year.

I'm happy the post helped you :)

It's just never been clear to me what HATEOAS is really supposed to be good for. Sure, a client can follow the links in an automated fashion, but how is it supposed to know what the resources actually are and which links it needs to follow, which resources it has to create or modify, to actually accomplish anything?

The general idea of returning links to related resources and/or actions is fine and good, but the rhetoric tends to go further, to ecompass claims like the API being "self-documenting" or amenable to a universal client. It always seems to me that this Big Idea of just presupposes the existence of a "smart" client that can really understand the links, one that there doesn't seem to be much sign of.

GitHub's API proudly notes its use of hypermedia and URL Templates in its responses, but I still have go read the documentation to decide what link I need to use, and what needs to fill into those variable slots in the URLs. The template doesn't do much for me that text in the documentation saying "these GET paramters are accepted/required" wouldn't just as well.

> Sure, a client can follow the links in an automated fashion, but how is it supposed to know what the resources actually are and which links it needs to follow, which resources it has to create or modify, to actually accomplish anything?

I've never understood this either. My API client isn't smart enough to follow links and write logic for me, so when they say "the client" can "discover", they must be referring to myself, and not my code? Well I'd much rather read documentation than click hyperlinks inside an API.

You can find smarter clients by Googling "hypermedia client". Also, O'Reilly just published a book on the topic: http://shop.oreilly.com/product/0636920037958.do

But would anybody really want to use a hypermedia client versus a regular client? Of course not. The regular client can be optimized properly at the level it needs to be to make the user's interaction smoother.

You are entering your comment into a general purpose hypermedia client for humans, why wouldn't you want a general purpose client for machines?

This is because the idea is that you would create a new media-type to represent your resource. It is this media-type definition that would determine what rel-types there are and how a client should interpret them.

for example, the spec for the HTML media-type that when a client sees a link with the rel-type "stylesheet" is should fetch the resource using HTTP GET.

As REST requires that media-types be registered the idea would be that we would eventually get a set of media-types that cover things like audio playlists, and how to interact with them.

So any "intelligence" required by a client would be baked into the implementation of the media-type processor. Instead of "client libs" for specific web services, you would have a general media-type parser/processor which could be re-used by clients of different web services to process common media-types.

But apparently individual client libs for each web service that overloads JSON is better.

> But apparently individual client libs for each web service that overloads JSON is better.

I mean, there are so many different types of resources and every API I interact with definitely invent their own. How often do you come across an API offering a playlist of music?

Yep. In theory, one could imagine various schemas at various levels of abstractions that allow the client to 'know' what's going on, automatically deriving it from the schemas.

I think this is also the theoretical promise of one approach to linked data/RDF.

In practice, I don't think it happens, and is not worth the conceptual and engineering overhead for the possibility of something that doesn't seem to be realistic to expect.

Indeed you need a client that is capable of mapping allowed methods and links to something useful, but it's not hard to imagine how one might go about this:

* For related resources (imagine a DB FK), instead of listing a uuid you point a link to the location of that resource. This means if the resource location changes the client doesn't need to be updated, it just uses whatever the new link is.

* Methods listed can drive available actions, e.g. imagine you have both mutable and immutable objects, a smart client could reason if PATCH is available then a UI element can be spawned allowing the user to modify this resource.

This is useful in a single client scenario whereby the API schema can be modified to a certain extent without the need to modify the client. It is also useful in a multi-client scenario (either distinct or versioned clients) for ensuring consistency between clients as you no longer need to ensure all clients are modified to use the new schema.

At least, this is how I understood it, I haven't tried it in practice.

How I interpret HATEOAS:

The client knows what and how it can access on the behalf of the authenticated user. Examples:

Representation for a user with no privileges:

    {
        "articles": [{
            "id": 123
            "title": "A title",
            "links": {
                "self": {
                    "href": "http://blog.com/articles/123",
                    "methods": ["GET"]
                }
            }
        }],
        "links": {
            "self": {
                "href": "http://blog.com/articles",
                "methods": ["GET"]
            }
        }
    }

Representation for a user who is authorized to add/edit/delete articles:

    {
        "articles": [{
            "id": 123
            "title": "A title",
            "links": {
                "self": {
                    "href": "http://blog.com/articles/123",
                    "methods": ["GET", "DELETE", "PUT"]
                }
            }
        }],
        "links": {
            "self": {
                "href": "http://blog.com/articles",
                "methods": ["GET", "POST"]
            }
        }
    }

This reduces the authorization logic on the client side.

How? At the end of the day you're still writing an "if" statement branching on a piece of data in the response. In a regular api it might be canEdit, here it's the http verb in an array.

To me the best advantage is that by following links, the client doesn't have to builds those links in the first place.

So the client will keep working even if a few months from now you want to change the link to something else. A simple example: if retrieving articles can be done by requesting this link "/articles", I can potentially change it in the future to "/v2/articles" and the client would still work.

The client may not have to build the links but it still has to be aware of their specifics. It has to know that /articles takes arguments X and Y, and that /v2/articles takes arguments Y and Z. You're trading building links to inspecting links. What advantages does this have?

One of the notions in HATEOAS as far as I can surmise is that we decide to build a common vocabulary describing our API and that becomes the fixed interface for interaction. It's not at all meant to imply that software clients can magically figure out what to do next (thought often a human with a reasonable API app could do so).

Imagine for example that we decided that whenever you request a comment resource from my API then I promise to provide a link in the response called "upvote" which you can follow to upvote that comment.

We have agreed to a fixed interface but left plenty of details flexible.

Maybe we are experiencing heavy load: let's stop sending the "upvote" link - the clients should understand that in its absence the operation is not currently available.

Maybe we have implemented some load-balancing system which redirects clients to `fiji.api-server.com` or `romania.api-server.com` based on their geo-ip data: those clients need only hard-code one top-level API URL into their code and all other URLs come through successive API responses. Load-balancing happens automatically and can even change throughout a single session because the client follows the links instead of building its own URL.

Maybe we are running some test or gradual rollout of a new API or URL structure; as long as we provide those links and references the clients can follow the right path without needing to know about the changes. It's possible that we ended up moving comments from `api-server.com/api/threads/1337/comments/42` to `api-server.com/comments/what-do-you-kow-joe` and this won't break any client designed to follow the interface instead of the incidental details.

For what it's worth I think very few APIs come reasonably close to this design and maybe few even have much need to. REST and HATEOAS become much more important when someone is publishing a public API that many third parties will consume and the ability to introduce non-breaking changes and server-side control of different specifics is important.

/my 2¢

> It's just never been clear to me what HATEOAS is really supposed to be good for.

Have you ever used a web browser? You know how the browser uses media-type to determine how to handle content referenced by a URI? That's what HATEOAS is supposed to be useful for: links identify and locate resources, resource type information tells you what kind of resource it is. The only out-of-band information you should need for an idealized REST API is information on the protocol used (e.g., HTTP) and information on the resource types (media types for HTTP) of the resources used.

(Really, if you want to understand any component of REST, its probably easiest to ask "what is this used to accomplish in the HTTP-based web", since REST is essentially a generalization of an idealized version of the HTTP-based web.)

Yes, but most people aren't making web browsers or similar.

Web browsers are extremely generalized: they display arbitrary HTML, submit arbitrary forms, download arbitrary images, download and run arbitrary JS/CSS.

Most APIs are intended for more specific uses than "all of HTML". Hypermedia is very useful for browsers, but I wouldn't extend that to say, my Imgur clone API.

> Yes, but most people aren't making web browsers or similar.

The question is whether people are making APIs to be consumed by general purpose tools, not whether they are making general purpose tools.

But, yes, there's a quite valid argument that REST isn't always the right architectural style for an API.

Which isn't an excuse to use the term "REST" for things that aren't REST, which just confused the issue.

Okay, either way. There aren't a lot of hypermedia-based clients.

Just guessing, if you had an API for creating documents for example, and you POST a request to /docs/ you'd get back not an just a single ID but a URL to /docs/<ID>. So then the client can operate on that resource and not have to compose it.

It can also be browse-able with a regular browser. If you visit it say with Firefox and go to .../api/ and the browser tells the backend it accepts text/html back, the service would return a list of sub-resources as proper hyperlinks. Pretty formatted json and so on. Might return a few items only not all if there are too many in a collection. After you see all sub-resources .../api/docs/ .../api/widgets/ etc. you navigate by clicking to any of those..

I have actually built that and it seemed like gimmick first but improved developer productivity quite a bit because the API is discoverable and provides live data (instead of just out-dated examples from docs that nobody updates.

Was it HATEOAS? I still don't know. But I knew it worked really nicely. Someone working on the backend could finish a feature and stick it in /api/newfeature/ a front-end developer would browser that get a sense of how it works, what data looks like and so on.

(Bonus points: figured a way to auto-generate docs from code comments from classes and modules and pushed them to the API resources, so now it is was self documented, and had live example and so on).

If you get back an URL to the document instead of the ID, then whenever you need to refer to that document, you need the whole URL. That means that it can't change, which I thought was one of the arguments for using HATEOAS, that you don't need to hardcode the URLs, and can "evolve" the API without breaking clients.

Good point. I think that might be a good thing too in the sense that you could support multiple API versions: by maintaining compatibility for older URL paths. Then clients could also crawl the hierarchy at startup or so to see if what they expect to be there is there or it changed. Maybe respond with a redirect if an older resource is accessed... In practice usually a v1 or v2 is shoved somewhere in the URL or headers.

This is a point which is I think overplayed by HATEOAS fans and under-appreciated by HATEOAS haters; using URLs as IDs makes it easier to evolve the API in many cases, but it doesn't make it completely painless to do so.

If you have an object which links to `/users/1/`, and want to change that URL to `/cool_users/1/`, what's the migration path?

Without HATEOAS, you need to update all your clients' code to now generate the new base URL `/cool_users/`. This means you'll need to version your API, so that old clients can continue to access the old-style endpoints in the transition period. (Note that for a business where your customers are making an API integration, this means you're imposing work on your customers).

With HATEOAS, you just need to update the URLs that are returned in your other endpoints. (Generally there is one well-known entry-point into your API, e.g. you return {"user_list": "/users/", ...} with your login token, for example). Now, assuming your clients were using `api.user_list`, that they received, they will without further modification fetch the `/cool_users/` endpoint, without requiring an update.

The one gotcha is if clients are holding on to the IDs of your API objects between API calls; in that case, you will break any code which expects to find those previously-returned members. But note, the worst-case here is that you need to version your APIs, which was the best-case without HATEOAS. In many cases you can get away with such a change without any client-facing changes.

Yes, correct. However, this relies on clients using your API in a hateos way - which they have to go out of their way to do: starting at /, reading the responses, navigating down only using URLs that you return, etc...

No clients bother to do this, in the real world - they just hard code/compose the URLs that they need to use. Why make extra http calls when you don't have to? Why parse all the json-hal (or whatever) to "figure out" which URL to call next, when you don't have to?

Even if most clients did this, you can't enforce it, so not all of them will, so some will still break when you change URLs.

This is why my HATEOAS APIs always return urls in the form "https://mysite.com/{SHA256 hash}", and a façade API looks up the actual path from the cached hash. Hardcode that, bitches.

Haha, I have definitely considered that path, but

1) It makes manual testing annoying, 2) I have a nagging feeling that if my users are "doing it wrong" then maybe the API is doing it wrong...

Also I've been playing with autogenerating client implementations using autogenerated swagger specs, and that approach is incompatible with an actual opaque linked API. It would be nice to have the best of both worlds.

That seems rather user hostile.

Very true - that is the best counterargument. However, we're still back to the worst-case here being the best case without HATEOAS, and well-behaved clients can still reap the benefits even if there are some misbehaving clients requiring multiple versions to be deployed in parallel.

There's a good question about how long you can cache those URLs for as well; it's a non-starter for a client to have to traverse the whole tree from the root for every request. So can I cache the responses for the duration of my auth token, and get a new root node as part of my re-auth?

If you go down that route, now you need to maintain two versions again during migration (but you do keep the ability for 'well-behaved' clients to migrate versions without downtime).

As the sibling comment describes, you _can_ enforce this by obfuscating your URLs, but I've not had the guts to do that yet...

Another approach would be to write great client libraries yourself, so that you know that the clients are consuming the API correctly.

> well-behaved clients can still reap the benefits

I don't think that the supposed benefits of HATEOAS actually materialise, in the real world.

(Note: I'm not advocating this approach, I merely repeat what I think I know)

Instead of hard coding the subject hierarchy in templates (like "/topic/subtopic/"), you define a document type that exposes target elements as links (typically with a "rel" attribute identifying the type of element). The client can then navigate the logical hierarchy without requiring this to match the physical URLs (think: federation across departments; instead of having a template "/api/{department_id}/people/{employee_id}", the company can expose a directory document that aggregates links "link rel='employee' target='https://departmentX.company.com/arbitrary/hierarchy/employee... and "link rel='employee' target='https://othercompany.com/api/users/Doe+John'").

The type of linked resources must of course be aligned; there's no magic involved. Instead of the client having out-of-band knowledge about the hierarchy, the client must have out-of-band knowledge about the used document types containing the links.

HATEOS, and REST in general, is a lot more useful when there are middlemen involved. If I have some link relation type named "api.myservice.com/rels/access-controlled-by" and some content type for authentication policies, then I can build a proxy between my API and clients that looks for links of this relation to resources of this type and automatically implements authentication checks. Instead of writing code to check auth rules in my API, I link to a resource that the auth proxy understands in a common format. This format can evolve over time without breaking the proxy due to content negotiation, and API services written in entirely different languages can still rely on a uniform implementation of authentication rules.

There's all kinds of other directions you can take this including quota enforcement, monitoring, auditing, and other resource-agnostic concerns. More radically, you can make these sorts of proxies reusable services that other people rely on to implement these behaviors. One of the primary motivations for REST in the first place was a standard interface that would allow for insertion of caches at arbitrary points in the Web without breaking everything (in the optimistic case at least). There's even HATEOS in the Cache-Control header, as the cache channels extension uses links to external resources to define the cache channels for resources

Use something like JSON-LD, where the payload contains hyperlinks to documents for each data property.

Your client then can machine read that documentation (which can have long-lived cache headers or be immutable, so your client doesn't spend it's whole time re-checking documentation).

There will be a set of definitions that the client understood when it was coded, and by checking a payload for matching meanings, it can consume the information.

If it finds something it doesn't understand, it could even try to take some compensating action (e.g. suppose it's a client which displays images to it's users, and it encounters a new type of image format. It could perhaps look up a registry of javascript canvas image renderers and download suitable code to display the new image).

FWIW, Roy Fielding's paper[1] on REST (where the word comes from) was very specific that a REST api is a hypermedia/hypertext based api, and that if it isn't hypermedia, it isn't REST. He clarified[2] this later on. This isn't from some internet toughguy, but the guy who literally coined the phrase. Sadly, his implementation was also all XML so gross, but you get the idea. HATEOS is just an implementation of a JSON based hypermedia api.

Me? I'm a fan of grpc for everything. HTTP/2 > HTTP1 and binary compressed protobuffers > deflate compressed (gz) json. http://grpc.io

[1] https://www.ics.uci.edu/~fielding/pubs/dissertation/rest_arc...

[2] http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hyperte...

If I get it correctly HATEOS is about the following:

Say you're building a standardised API (with many consumer implementations _and_ many provider implementations all talking to each other). In that case making stuff like the layout of a url or even the protocol over which the data is retrieved part of the specification is putting a needless strain on your API providers. If an API provider uses a shared webhost or a specific framework he might not be able to follow that structure, or it might be harder then necessary.

If instead, your specification only specifies the document layout (Roy Fielding calls it the "media type") then it's easier for people to implement your api. People usually have full control over the document contents.

If you don't design your API like that it's not a REST Api according to Roy Fielding. If your API is not going to be implemented by many providers and many consumers all talking to each other in countless permutations then there's not much use in following all of REST[1]. (And also not much use in calling your API a rest api)

Of course the word REST now means something else then Roy Fielding intended for it. I have no problem with that. I believe he did at some point.

[1] I don't remember the url or the exact quote, but I believe I read a piece by him where he said literally that REST as he described it is only useful for the many2many consumer/provider use case. I might be mistaken.

It even cooler than that: You only have to standardize the types and meanings of the links, the actual implementations can vary in the locations of the endpoints, but the structure of the documents can too, and the client(s) will be smart enough to figure out that on this server they need to retrieve /users/mike/products/, and on that server they need to get /userproducts/mike/all/, just through the fact that the representation of the user (also probably retrieved from different locations) on each server has a link labeled rel="all products" pointing to the right place, regardless of what else was included or omitted from the representation, so you can have interoperability on whatever is standardized even if implementations of both clients and servers have different capabilities they expose.

So, as long as you standardize the semantics and relations, the URL layouts and the document layouts can be quite different, in the same way that you know when looking at a page what the button or link labeled "Home" will do, regardless of the exact location on the page or the fact that on one server it takes you to /, on another to /index.html, and on yet another to /main/home/ .

> In that sense, REST in practice has always been just RPC without a clear spec for procedure call like XML or JSON RPC.

HTTP, even when used without hypermedia, still has interesting features that are not found in traditional RPC. For example, you can’t just stick nginx in front of your gRPC server and tell it to cache stuff. You can’t tell your Thrift client to retry all idempotent requests automatically (without enumerating the functions that are idempotent). Not to mention upcoming things like HTTP/2 push and 103 (Early Hints).

I like to think that HTTP is less of a leaky abstraction over the network.

Web frontenders in my experience want REST APIS that are semantically thought out (aka "nice") rather than compliant with some spec.

...HATEOS, IDGAF if //api.me.com/article/list uses 404 to mean there are zero articles returned from an otherwise valid resource.

Just yesterday, I asked a back-ender to put the total_pages, page_limit, and current_page on their endpoint.

BE: But you should be able to use my "self", "next", "prev" links?

FE: Well the thing is, I want to generate and render links for every page

BE: Oh, I see...

As I see it, HATEOAS by providing API structure dynamically depending on application state, restricts the application to non-concurrent use and full backward compatibility, because application state perceived by the client and the actual state might diverge in concurrent use (either concurrent clients or state change by code upgrade). It is impossible to build a client (which does more than query a resource in a loop and simultaneously does the right thing) that will magically know about newly available link if application state changes between getting the resource, making a decision and following a link based on that decision. Which clearly breaks the very intent to make the API self documenting.

Taking Wikipedia example [1], how can a client know whether the application stopped supporting withdrawals altogether (it is dynamic after all) or is it the particular account it is querying? What happens when account is overdrawn between a client querying state and attempting a withdraw? When should a client stop expecting returned endpoint to be still available? How do you document possible outcomes?

Another example of HATEOAS not being thought through is the self link. If a client is querying application through proxy (e.g. forwarded port to directly unreachable destination or some kind of aggregator application), the self link becomes incorrect. Should a client know to magically rewrite the link or treat it as a redirection? If the application is reachable by various paths (domain names, IPs) it must know the path client took, to return correct self link even without a proxy.

HATEOAS restricts application to a set of very specific use cases with subtle traps to fall into. I see no way to avoid both these traps and need for documentation, which, in my view, defeats the purpose. Your mileage may of course vary.

[1]: https://en.wikipedia.org/wiki/HATEOAS

An interesting design question arrises around nested resources. Google in this doc buys into deep nested structures, e.g. `//calendar.googleapis.com/users/john smith/events/123` (from [1]).

I think this pattern is unambiguously sensible when the child objects are strictly scoped under the parent.

But it's less clear how to represent resources that are shared between multiple parents; for example, what if event 123 can be referenced under another user's API resource as well? If we permit `//calendar.googleapis.com/users/bob/events/123`, now we have multiple URLs referring to the same object, and things can get quite tricky in the implementation.

Django Rest Framework strongly discourages (and makes it quite hard to implement) nested resources, FWIW.

I've found that a policy of only permitting one level of nesting seems to be a good balance for shared objects, e.g.

`//calendar.googleapis.com/users/john smith/events/` returns:

``` [ { url: "/events/123"}, ... ] ```

Interested to know how others have solved this problem.

[1]: https://cloud.google.com/apis/design/resource_names

Here's one way to tackle it, if behind your REST API is an SQL database:

  /schema/table/key

So:

  //www.example.com/calendar/events/123

To address many records, like all that belong to Bob, use the query string instead of purely the path:

  //www.example.com/calendar/events/?user=bob

This is in my experience the `standard` design; it does give you a lot of freedom to change what filters you allow, and to stack them. Nobody's going to get fired for this design, and there's a lot of prior art around it to draw examples from. It also has the benefit of keeping the API surface small and clean.

But it makes it a bit weird to do HATEOAS; you _could_ do `GET Bob => {events: "calendar/events/?user=bob"}` -- but then you're hyperlinking to a search and not a resource.

It also tells less of a narrative in the structure; `user=bob` is just another filter that you can use to apply to the events set. But we get a chance to describe the shape of the data a bit more if we choose to declare an intermediate resource (/users/) and attach some links to it (=>/users/bob/events).

Now, if there are ten ways that you need to slice your `events` set, and ?user=bob is but one of them, then scoping a sub-resource /users/events/ isn't that useful/descriptive.

As an aside, I think this is where HATEOAS is nice; it makes it very easy to navigate an API as a developer, see what actions are possible at every node, and hopefully learn the intent of the author of the API without having to chew through a set of API documentation. Django Rest Framework's API browser is a great example here.

I prefer the bare minimum approach. That is, if you NEED the user resource to access the event (eg the primary key is like [user ID, event ID]) then nest it in the url. Otherwise if an object has its own ID and could otherwise be accessed independently why not just do so?

The root of this question is: how do you handle M2M relationships in your API?

As described in https://cloud.google.com/apis/design/design_patterns#get_uni...:

  //calendar.googleapis.com/users/-/events/123

Hadn't spotted that, it's an interesting syntax and I quite like it.

But it doesn't cover the case I'm referring to; in their example you MUST return the fully-qualified URL:

> shelves/shelf713/books/book8141, not shelves/-/books/book8141

This precludes there being multiple shelves with the same book (sensible in their example, since a book has only one shelf, but not in the example above).

While sometimes this does get tricky with resources that are clearly owned or associated with multiple things, often times a resource only has one real owner and it is unambiguous. It does make moving things a bit weird though, as moving a book between two shelves makes it have a different fully qualified name.

Agree; put another way, sometimes you really do need a M2M relationship.

What they describe is not REST. Nowhere in this document mentions hyperlinks, a strict requirement of the REST architectural style.

The best analogy would be a simple web page, which usually contains hyperlinks that a client can follow to discover new information. Unfortunately, web developers' understanding of REST ends with HTML, and they re-invent the wheel, badly, every time they create an ad hoc JSON-over-HTTP service.

There is a standardized solution for machine-to-machine REST: JSON-LD [1], with best practices[2] to follow, and even some formalized specs[3][4]. To Google's credit, they are now parsing JSON-LD in search results, which is much nicer to read and write than the various HTML-based micro-data formats.

On a related note, REST has nothing to do with pretty URLs, naming conventions, or even HTTP verbs. That is to say, it is independent of the HTTP protocol, but maps quite naturally to it.

[1]: http://json-ld.org/

[2]: http://json-ld.org/spec/latest/json-ld-api-best-practices/

[3]: http://micro-api.org/

[4]: http://www.markus-lanthaler.com/hydra/

This is an API guideline, not a strict REST one. Also it has become very clear to me everyone has a different interpretation of what REST can or should be. We all are quick to forget that the actual acronym stands for "representational state transfer" which is an abstract concept and therefore can be implemented in many, many different ways.

>it has become very clear to me everyone has a different interpretation of what REST can or should be

This is very true, the term has been abused so much that it has no relation to its original meaning.

We can loosely define REST by one of its key qualities: hypermedia, that is a lot easier to say than HATEOAS. By focusing on this distinction, it rules out 99% of APIs in the wild.

On top of that, hypermedia media types and linked data are important for an API to be self-documenting, so there is a rather high bar for developers to implement.

It's heavy 'REST' oriented.

A Java guideline for API's would like completely different.

The guide actually explicitly says it is focused on gRPC.

How is:

    {
      "@context": "http://json-ld.org/contexts/person.jsonld",
      "@id": "http://dbpedia.org/resource/John_Lennon",
      "name": "John Lennon",
      "born": "1940-10-09",
      "spouse": "http://dbpedia.org/resource/Cynthia_Lennon"
    }

Better than:

    (http://json-ld.org/contexts/person
     (id http://dbpedia.org/resource/John_Lennon)
     (name "John Lennon")
     (born 1940-10-09)
     (spouse http://dbpedia.org/resource/Cynthia_Lennon))

Note that the latter specifies a single canonical representation[1], which can be hashed and used for comparison, while the former does not (unless one specifies e.g. alphabetic ordering of properties). The latter format may also has a standard specification for transmission e.g. as a URL parameter[2].

And it's obviously a student-level exercise to write a validator, spec or grammar for the latter format, while the former is rather more open-ended.

Friends don't let friends JSON.

[1] Which HN sadly turns into a linked mess — Base64-decode the transport representation[2] if you'd like to see the canonical representation.

[2] {KDM0Omh0dHA6Ly9qc29uLWxkLm9yZy9jb250ZXh0cy9wZXJzb24oMjppZDM5Omh0dHA6Ly9kYnBl ZGlhLm9yZy9yZXNvdXJjZS9Kb2huX0xlbm5vbikoNDpuYW1lMTE6Sm9obiBMZW5ub24pKDQ6Ym9y bjEwOjE5NDAtMTAtMDkpKDY6c3BvdXNlNDI6aHR0cDovL2RicGVkaWEub3JnL3Jlc291cmNlL0N5 bnRoaWFfTGVubm9uKSk=}

The answer to that is really simple: JSON is a widely supported standard in many platforms and languages. Whatever the latter is, is not.

What's better than writing a parser for the latter format, is not having to write anything and just using a library, which exists in practically every language for JSON.

I'd argue, choose whatever format is best for your case. If its easy, chose a binary format for small footprint and a human readable format. But chose formats that are semantic and are able to drive the client. That's what the "R"epresentation is all about in "REST".

Ever heard of LISP, or S-expressions?

Of course, I even write some hobby projects in Common Lisp.

There are data interchange formats that use S-Expressions, namely EDN[1]. But JSON remains the most popular format for its widespread support, and its few data types map to most languages.

[1]: https://github.com/edn-format/edn

I personally like collection+json as a response type for REST. I will say that my own efforts at using a hypermedia-containing response type for a REST api shared across several teams have been mostly wasted effort. Consumers of the api treat the responses as plain-old JSON which somewhat kills the advantages of having a proper REST response in the first place . . .

> What they describe is not REST. [...] a strict requirement of the REST architectural style. [...]

You have a word "REST" for which you are apparently granted access to Plato's "true" definitions, which enables you to tell me that REST requires hyperlinks, but not naming conventions or HTTP verbs.

I reject your definition.

Go ahead and use that word "REST" however you like. I will continue using it to describe what you consider to be "ad hoc JSON-over-HTTP services".

Sure, I've read Fielding's dissertation.[1] I think stateless, cacheable, layered systems are a great idea. I think "code on demand" is (usually) a stupid one... even if it does turn out to work surprisingly well for web browsers. But none of those matter.

I work with people who build "ad hoc JSON-over-HTTP services". They spend hundreds of millions of dollars building ad hoc JSON-over-HTTP services. They call them "REST" services.

I have to talk to these people, so I call them "REST" services too. Because I'd rather build something useful than spend time telling people that they're using a word "wrong", when the only real meaning of a word is whatever it will bring to mind in the person you are communicating with.

[1]: http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm

I am not the authority on what REST is. That would be Roy Fielding, who has explicitly stated that hypermedia is a requirement[1]. So go ahead and tell Mr. Fielding that his definition of REST is incorrect.

I am well aware that REST no longer means what it originally described, which is why I think it should go by another term that is not burdened by being a marketing buzzword.

[1]: http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hyperte...

Don't worry, the term HATEOAS won't become trendy any time soon.

It's far too late for that. REST as you describe it is not commonly known as REST. People refer to it as HATEOAS or "real REST" or "hypermedia REST". Just saying REST without qualification refers to resource-oriented json based apis.

"Hypermedia APIs" seems to be the up-and-coming term.

> So go ahead and tell Mr. Fielding that his definition of REST is incorrect.

Not "incorrect", but I'd be happy to tell him that his term has been co-opted by the programming masses to means something vaguely related to the original meaning but less precise. I suspect Mr. Fielding already knows that.

> I am well aware that REST no longer means what it originally described, which is why I think it should go by another term that is not burdened by being a marketing buzzword.

I agree there. I think changing how the masses use the term is a lost cause. (Consider the incredibly hard-fought battle to reclaim the original definition of "hacker", which after decades did succeed in establishing it as a secondary definition. That's the most successful case I've ever seen.) So I think ryeguy has it right: call Fielding's definition "HATEOAS" "real REST" or "hypermedia REST" or something.

Wow, it seems linguistic prescriptivism has a stronger foothold in engineering domains. Probably fewer social class issues.

Well you sure confused the heck outta me.

---

It is really so hard to say "HTTP" instead of "REST"? "Regardless" instead of "irregardless"?

Very interesting read! I like that GOOG is pushing gRPC more on their own services. I've been a gRPC user since Sep/Oct last year, and it's made developing for Android, Node.js, JVM, Python more pleasant from a networking perspective. The ease of just moving logic from Node.js to a Java gRPC server, and then redirecting the HTTP2 proxy to the right place, has been awesome.

I've started teaching some people in the team how to use gRPC, and we're def going to be using it where permissible on client projects.

Please drop fixed headers from web pages. If you want easy access to the top of the page use anchor links instead. On a laptop headers often take a big chunk of available screen. It just pisses me off every time I see a page with a fixed header. All your reader aren't using imacs...

This is a marketing website. I'm sure they A/B tested the fixed header and it probably converts better than otherwise.

This isn't a marketing website this is the documentation website for google cloud. I'm logged in right now on google cloud and it's still displaying that header.

Yeah? Who's paying google the big bills? The people who are looking to "CONTACT SALES," which is conveniently a link in the fixed header.

Extremely annoying on mobile. So much that it made me not want to read the page anymore

And it tends to break using Space or Page Down to advance.

I really wonder if hipsters ever actually read web pages, or if they just load them, look at them and then go back to discussing the merits of their fair trade, artisanally-roasted espressos.

I am curious if anyone went to GraphQL without regrets?

We've been using GraphQL for everything since late 2015. All recent code is GraphQL-first, and all old code is proxied by a GraphQL layer in front of it.

Our application helps BigCos to understand if they pay people fairly and to run smart pay reviews. It's a relatively small codebase, ~100k LOC, but it's essential complexity is in managing and connecting dispersed data about employees and markets. GraphQL allows us to represent the natural links within this data, then the app frontends can present whatever business information is helpful in that page/sidebar/widget/card without separate endpoints.

With REST, we had the same problems with every feature: over/under-fetching, can't express relationships well, and can't evolve the schema easily. When we tried to work around these issues (e.g. "v2?fields=a,b,c"), we ended up with a poorly implemented subsection of GraphQL that's not benefiting from Facebook's experience. To compare to the world of databases, I view REST as a Key-Value protocol and GraphQL as an SQL with joins and functions. If all you need is to lookup a document, don't overcomplicate it. But if you need to express relations, you don't want to do that in userland.

The only advantage of REST is using a widely known standard with rich tooling and well-published "best practices" (that just try to work around REST limitations).

REST describes relationships very well, via hyperlinks. You navigated to this page via a hyperlink. If your API doesn't do that, it's not REST, this is what people usually mean when they point out that an API isn't complying to the REST style.

REST describes relationships just fine. Now return a list of 100 documents that each have a list of related comments. Your client just needs to request

  GET /documents
  GET /documents/1/comments
  GET /documents/2/comments
  GET /documents/3/comments
  GET /documents/4/comments
  GET /documents/5/comments
  ..
  GET /documents/99/comments
  GET /documents/100/comments

Easy, right?

Easy mode: GET /documents?include=comments

Think of how this can be done on a web page. The documents page has a link to follow, "include comments", which links to "?include=comments". The exact query doesn't matter, what's important is that a client can discover new information without needing any out-of-band information.

That's where we started.

The next natural step will be - "Thanks for the comments, now the user needs to sort them by votes/date, ok?" and the URL starts looking like a query anyway, so you're on the way of re-implementing GraphQL. Then do you send avatars/timestamps/changeflags for all comments, or only on main body in this endpoint? Now you have the over/under-fetching problems.

The step after that will be "Oh there are 300 of them - I just want the top 3 comments first, and a pager for the rest ok?" and you simply can't do that with the structure REST mandates.

>the structure REST mandates.

There is no structure that REST mandates, if there were it would be a specification and not an architectural style. There is nothing that says that you can't query for that:

GET /comments?sort=votes&limit=3

The exact string doesn't matter, what matters is the client can discover this. How HTML does this is by generating query parameters based on form inputs.

REST limits my knowledge only to the primary key of the relationship.

Given a trivial question like "here's a user, I need her friends and their countries of birth" and the REST answer is a separate endpoint or an O(n) operation on the client because these are _separate resources_. You want the country flag too? I'm sorry, I can't support that requirement.

With GraphQL, it's -

    user(handle: "daliwali") { 
      name,
      friends {
         country {
           name, population, flag
         }
      }
    }

No separate endpoint. You want to show to the client how many dogs those friends have, and if any of them play frisbee? No problems, and no backend engineers involved.

This is why I said REST is a key-value protocol like in redis. With redis you can either embed a small objects (which is not a relationship) or keep a PK of the relationships (which means many queries). The more consumers your API has, and the higher the latency, the more expensive either of those choices becomes.

You are correct that in a REST system, every resource has a "primary key", that is the URL.

Where you are wrong is that REST doesn't mandate that a resource can only be accessed by its "primary key". There is nothing stopping me from requesting the following URL:

GET /users/daliwali?fields=name&include=friends,friends.country

Any client would be able to follow that link and get something out of it, whether it's JSON or HTML, without needing specialized tooling such as a GraphQL client. A hyperlink makes that query widely accessible and interoperable with any HTTP client.

We're reasoning about a simple case, however. GraphQL is recursive, and HTTP parameters are generally not. You can use them to wrap a recursive language, but at that point you're just using the URL as a transport layer for the recursive language, and if the language is expressive enough it won't fit in a GETtable URL anyway... Past a threshold of complexity, you get into wanting that GraphQL client.

In reply to fixermark, there is also nothing stopping you from very complicated queries in REST. There is not even a requirement that the server must respond immediately. For example I could request:

POST /queries

With some raw database query as the payload (please don't actually do this) and the server could respond with HTTP 202 Accepted, meaning that it's going to take some time to process, meanwhile check back at the URL in the Location header when it's finished.

REST does not mandate any upper bound on complexity, that's up to you to decide.

But at this point, it's so different and you've added so much work to it, I doubt anyone would recognize it as originally REST. You're basically rewriting GraphQL yourself. And sure, you're allowed to do that, but why?

>it's so different and you've added so much work to it

With web pages, it's standard and effortlessly handled by the browser. Given a form with inputs specified by the server, a client sends a request with media type application/x-www-form-urlencoded or multipart/form-data. There is a vast number of implementations for practically every language and platform. A machine client can send a form too, or JSON for that matter.

You have it backwards, Facebook is attempting to rewrite web standards by themselves. And that is undermining the open web.

> In reply to fixermark, there is also nothing stopping you from very complicated queries in REST.

What I believe others are getting at is that those complicated queries themselves need to be expressed somehow even within a REST request. GraphQL provides a convenient mechanism for doing so.

In theory you can even do GraphQL RESTfully, with each REST endpoint exposing its own schema. But in doing so it quickly becomes apparent that it would be easier just to forego REST conventions and expose a single consolidated schema.

Speaking for my own GraphQL experiences, on one project we had a GraphQL query generating reports that was around 50 lines long, with half a dozen fragments. Flattened out to match the approaches you've suggested for POST data and GET params it would have been hundreds of attributes. It's unclear that there's any open standard that would have provided as simple of a solution for expressing and executing those requests.

that's almost exactly what a graphql get looks like

    http://myapi/graphql?query={me{name}}

http://graphql.org/learn/serving-over-http/#get-request

> Given a trivial question like "here's a user, I need her friends and their countries of birth" and the REST answer is a separate endpoint or an O(n) operation on the client because these are _separate resources_.

Actually, I think that the answer following the REST architectural style, using HTTP, and not resorting to a custom HTTP method (which, actually, consistent with the REST architectural style would arguably be justified in this case), the correct approach would be to POST a representation of a resource representing the query to an appropriate query endpoint (which could be the user endpoint for queries specific to that user, since the query could reasonably be seen as a subordinate entity to the user to which it applied), and then GET the result of that query.

(In fact, the given GraphQL could well be the representation of the query.)

Ideally, HTTP needs a generalized safe method (like GET) that takes a payload (like POST) named something like SEARCH (there are HTTP-based technologies with domain-specific versions of this, but no generally accepted representation-neutral version) so that this can be, when the server architecture supports synchronous exchanges, a one-round-trip process.

> This is why I said REST is a key-value protocol like in redis.

But this is simply false. Its trivial to make key-value protocols that follow REST (especially the REST-minus-HATEOAS that is common these days), but REST isn't limited to that, or even specialized to it.

Any idea how it compares to oData?

Sorry, no experience with oData so can't give a useful comparison. One obvious difference is that GraphQL requires a structured query, which will always guarantee the structure of the response data. It's like static typing for business data objects. oData seems to embrace REST, so when I request a resource /person/james I have no assurances about what the response will look like. What fields are present? Is "email" a string or an array of strings? What's deprecated? RTFM. GraphQL is explicit so I know exactly what data is used in what views - makes change and debugging that much easier.

OData requests and responses have schemas, so you definitely have assurances on the field types. You _do_ know that email is a string or array of strings. http://docs.oasis-open.org/odata/odata/v4.0/odata-v4.0-part3...

Do you expose your GraphQL endpoint to customers, or just use it internally?

We expose GraphQL to two high value customers, but the pains of having a stable public API are so great that I wouldn't recommend it to any startup without a damn good reason.

Seems pretty good. Specifically this part of the guide is pretty well written: https://cloud.google.com/apis/design/resources. One thing that is surprising to me however is that their is no mention of using HTTP Status Codes in responses.

Thanks for the comment. The error handling chapter will be published in a few weeks. For now, you can reference https://github.com/googleapis/googleapis/blob/master/google/....

Disclaimer: I am one of the co-authors.

Canonical error codes used by Google, mapped to HTTP error codes: https://github.com/googleapis/googleapis/blob/master/google/...

Well, I can tell you from using their API's that Google doesn't necessarily use them.

I don't recall that specific scenarios, but over the past year I learned this with the maps API. As long as you're hitting a real endpoint, then it will return 200 - even when there are errors and it should clearly return a matching http status code.

I suppose it so the same API could be implemented in any given protocol, however, I don't think that is useful in many cases.

You might have been hitting the old JSON-RPC endpoint of that API, which has to return 200 to follow the standard. Or a gRPC version if there's one already.

Otherwise HTTP status codes for errors are used, and standardized internally.

Many Google APIs were created before this guide. New APIs published at https://github.com/googleapis follow this guide. Having the same API available via both REST and gRPC is very valuable, as gRPC often provides 10x performance.

Using HTTP status codes in your responses is a trap. It conflates the API transport with the actual semantics of the API. The goal of HTTP error responses is to say that something went wrong in the transport layer. The goal of API error responses is to say that something went wrong in your service. For example, your HTTP REST server may be perfectly fine, but your back end DB may be misbehaving. Having separate API level error responses, for example an explicit field called "error" in your JSON response, I consider to be best practice. Frequently your client needs to know the difference, for example to determine what kind of error to return to the user or what retry strategy to use.

In typical HTTP REST services this transport/API error split makes it really easy to create client code which only needs to check two conditions - if the HTTP response code is 200 or not, and if the error value is set or not. You also don't have to shoehorn your error handling into the very limited set of errors provided by the HTTP protocol.

The other real-world advantage of this is that when you outgrow HTTP as the transport protocol for performance reasons this makes porting the API really easy to, e.g. protobuf RPC, or even raw TCP. The error is already defined as part of the API and you don't need to rewrite all your client code to deal with mapping multiple HTTP response codes to your new transport. It's good future proofing I've seen pay off in a at least a couple of real-world cases.

Bottom line - your server should always return HTTP status 200 and a separate API error response.

There's also a reasonable debate to have about whether non-error responses should also include an explicit "error" field with some default OK value. There may be good reasons to leave it out, e.g. if you want to save bandwidth, but I consider that a fairly insignificant point. For consistency my APIs always return a default OK error field on non-error responses - your mileage may vary.

That is great in theory but it throws out all of the deeply rooted tooling that reports and aggregates http status codes across the service architecture. It is nice to be able to tail the apache log and see the status. In your situation I would need to have an additional system that uses application details for similar purposes.. you can do it as you describe but there is a cost! It is also against convention, which increases the custom tribal knowledge that people working in that system have to acquire. By the time you "outgrow" http I assume you'd have custom metrics and logging/tracing, but I would not suggest it as a good place to start.

HTTP has some very useful status codes, which convey standard conditions found in most apis: 200, 201, 400, 401, 404, 403, 406, 429, 500

Also note that not all APIs will be used be developers. Often it's someone less proficient with programming, and you can't rely on them to check the content of the return message. Help them help themselves by making curl (or whatever) bitch when there is an error.

It takes a bit more work to design an API that works over multiple transports, but a good framework, such as Servicestack.net if your in .Net land, mostly does it for you. By advocating 200 for all responses you're basically reverting to SOAP and WCF (Windows Communication Foundation).

Each to his own though, and for internal stuff, it might make a lot more sense.

Thank you all for the good responses on the error code issue. I particularly appreciate the ops view on this, e.g. the existence of much tooling around monitoring HTTP codes. Since I'm a pragmatist I'm not going to defend my point of view too vigorously, but concede that there are other considerations, particularly when designing services to work at scale. Everything breaks at scale, including many things we believe to be wise and true :)

Context: Friendly banter

>> Bottom line - your server should always return HTTP status 200 and a separate API error response.

That is a very absolute statement from a pragmatic guy, asking people to concede viewpoints other than their own :)

I'd be interested in learning how status codes other than 200 cause problems at scale though?

The problem with things like 404 is then you need to differentiate between

"Hi, this is the application and the object/document/whatever you are looking for was not found"

with

"Hi, this is the server and the api endpoint was not found"

An API I use returns a standard apache 404 error page when the item you are looking up doesn't exist. If the API endpoint was renamed my code that consumes it wouldn't have any idea anything was wrong.

I mean, if the API endpoint is renamed, they have broken all their clients.

Yes, an additional 4xx code to help differentiate between api endpoints and resources would be nice.

I didn't mean to imply that HTTP status codes could stand alone as error messages, so for a 404 error I'd also respond with more data, to help the user identify the issue.

Yeah.. The bigger issue with this app is that it responds with a default apache 404 page instead of a 404 code + its usual xml response.

I think that this example illustrates why parent mentioned that this is a trap.

Our experience, from having gone that route internally and externally (e.g. with JSON-RPC), is that it ends up not paying off to have that separation.

From the client side, developers shouldn't need to worry about what piece of your stack was at fault. They want to know if it's your fault or theirs/their user's. And if it's their fault, how to prevent it or fix it. Both HTTP and gRPC are flexible enough to encode that information.

From the server side, it complicates monitoring in practice if errors are propagated encoded in the response body of a "successful" RPC. With reverse proxies and other things in place, the complication increases, and many people get unhappy.

W.r.t. migrating clients to other transports, the client library should hide that (we haven't done that always right in the past, but we do now).

You are just plain wrong. HTTP was designed to access and update named resources. Those status codes correspond to the status of that resource so are totally appropriate no matter if the resource is a static HTML file or a database driven JSON response. I would hate to consume your APIs over HTTP.

What about status=ok|error[|partial]? how do you feel about this? Given its value you may have either result or errors attribute.

> The goal of HTTP error responses is to say that something went wrong in the transport layer.

Huh? What networking model are you working on?

HTTP is application layer.

TCP is transport layer.

IP is internet layer.

Ethernet is network layer.

Fantastic Read!

But I am still looking for some books on good API-Design, anybody has any recommendations?

If you don't mind Java, Effective Java by Josh Bloch has good API design material. Josh designed collections API in Java

very good read, but not really about web/http/rest API's.

Not sure what level you are looking for but Build APIs you won't hate was a great resource to me.

https://www.amazon.com/Build-APIs-You-Wont-Hate/dp/069223269...

Check out Apigee's ebook on good API-Design: https://pages.apigee.com/rs/apigee/images/api-design-ebook-2... disclaimer: both are my employers

What is current consensus on client libraries? Braintree for example requires that you use their client libraries where as Stripe makes them optional. With Google's gRPC thing I can definitely understand using libraries for performance. Otherwise, isn't making simple REST calls without custom libraries sufficient for most uses? Or if you want a library, something generic like Unirest [1]?

1. http://unirest.io/

Client libraries can be helpful and sometimes reduce a lot of plumbing/boilerplate that you would end up writing on your own (eg: authentication, paging).

Most environments -- Ruby, Python, PHP -- have minimal HTTP clients in standard libraries however they all have capable third-party libraries. Unirest as you mention, but also Requests for Python or Guzzle for PHP.

Product-specific clients generally build on these third-party libraries or go with the standard libraries instead. The product-specific clients generally offer more comprehensive error handling, for example if an API relies on arcane error codes that wouldn't immediately be obvious to an end-user.

Might be helpful to see Braintree's rationale for why they don't publish docs for plain external REST: https://www.braintreepayments.com/blog/when-rest-isnt-good-e...

I don't necessarily agree with all the points, but I can see why they made the decision they did.

I've read it several times and found it uncompelling.

I've used the simpler GCP APIs (like the Machine Learning ones) with direct REST calls. I've also been forced to use the REST API for things like Google Sheets because the client library documentation was so confusing.

For more complicated services, using a client library makes sense. Why reinvent the wheel?

With gRPC/Swagger/OpenAPI/etc you can also generate your own client stubs if you need to.

IMO, if you require a client library, there better be a really good reason...

(I work at Google Cloud, and often work with the API/libraries team. Opinions are my own)

Isn't building a client library for a REST API the definition of reinventing the wheel?

If your REST API is documented in OpenAPI/Swagger spec, then you can use Swagger Codegen [1] to generate API clients (Java, C#, PHP, etc), server stubs (e.g. C# NancyFx, PHP Lumen, Python Flask, etc) and API documentations.

[1] https://github.com/swagger-api/swagger-codegen

Step 1) document the endpoints enough that outside developers can write their own clients.

It took quite a bit of work for me to get a native Clojure client working to connect to the google cloud SDK. That was after wrestling with jar-hell around gRPC and calling the Java client from clojure, which is decidedly not pretty.

In these situations the best thing you can do is man-in-the-middle the HTTP requests from an existing client.

That said, I've only ever had to do this when reversing private mobile API's. I can't believe a REST API from Google would be missing documentation of the raw HTTP endpoints?! That should be the first documentation of an API, before that of any specific client implementations.

Or get a pre-existing client for a non-compiled language. Chances are there's a Python/PHP/Ruby/Perl implementation, and those shouldn't be hard to pick apart.

For future reference, here's the HTTP/REST documentation for the Cloud APIs: https://cloud.google.com/apis/docs/overview

E.g. for the GKE API: https://cloud.google.com/container-engine/reference/rest/

(For people that rather use an existing library, this lets you pick one of 7 languages and start from there: https://cloud.google.com/docs/ )

Yes, but that doesn't deal with authentication. My problem was getting the JWT stuff working. The solution ended up being: https://gist.github.com/arohner/8d94ee5704b1c0c1b206186525d9...

Ah, yeah, coding the OAuth2 flow in a new language (vs using an existing library) is tricky, at the very least because OAuth2 itself is complex. The best docs I know for that are here https://developers.google.com/identity/protocols/OAuth2Servi... (we should link to it from the docs of the Cloud APIs).

From a glimpse, what's said there matches what your code is doing, so thanks a lot for sharing it!

They didn't mention one very important thing - querying only required data and making connections between resources. For example, you need to download some git commits with user profiles. User is a different resource than git repo. How we can request such data in one single request? Then you will need also to load referenced issues (if present) that is implemented as different resource.

GraphQL solve this problems in a very nice and flexible way.

If the data is different there is very little need to ask in a single request. Just send two parallel requests. Of course in some cases getting a list of objects is cheaper then multiple requests but that is only if the objects are "related" and stored together.

However I do agree that GraphQL is great for a lot of use cases.

Neither side supports the case of [promise chaining](https://capnproto.org/rpc.html), where one or more resources can be used to look up further resources in a single round trip. Each style has tradeoffs.

  HTTP Method DELETE. Payload: empty.

I know DELETE is not supposed to have any payload, but using PATCH is awkward if you have to delete multiple resources based on a query or a filter. You need to specify a 'delete' action as part of PATCH request which means the payload model has to be different. Just awkward.

You face the same problem if you want to just list multiple resources based on a query or a filter. The solution isn't to send a GET with an HTTP body, which is counterintuitive to most people, and many proxies just drop. It's to make your filter part of the URL (as a query parameter). Same with DELETE.

A bulk deletion isn't defined as part of their "Standard Methods" .. For stuff that doesn't fit the standard methods, they have this page on custom methods: https://cloud.google.com/apis/design/custom_methods

That said, I'd implement a bulk deletion in one of three ways:

:::: ONE

DELETE /thingies/id1,id2,id3

When all is ok, the response is a 200. However, if one of the deletions fail, how you handle the response is more complicated.

:::: TWO

POST /thingies/bulk_delete

and have your ID's listed in the body. Still same problem with handling the response.

:::: THREE

POST /bulk

Instead of implementing a bulk deletion at all, think about how you can implement "bulk requests" as a higher level feature of the API.

So you can transport a bunch of delete's as part of a single HTTP request, and get back a bunch of response codes packaged into a single response.

  POST /thingies/bulk_delete

Exactly. That's the approach we've taken. It has worked better for us than sending a delete action via PATCH.

My biggest pain when designing a rest API is a standard authentication method that won't drive me crazy. So far i've always used 3rd party modules to implement different kinds of authentication but I never quite understood it in depth.

Apart from HTTP Basic Auth, but please don't use that.

> Apart from HTTP Basic Auth, but please don't use that.

What's wrong with basic auth with HTTPS? You can delegate authentication with OAUTH and then use OAUTH for authorization but authentication still has to be done somewhere.

I think Twillio API still uses or used basic auth via HTTPS.

So does Stripe. There is nothing wrong with basic auth for api tokens so long as you're using HTTPS.

> What's wrong with basic auth with HTTPS?

The only thing wrong that I can see is that it's 2017 and the browsers still don't have a good (indeed, AFAIK, any) UI for logging out.

Or for staying logged in across sessions (or not). But it should be fine for an API.

Basic auth is great. It lowers the barrier of entry, to anyone with a browser. Not all users of APIs are developers.

https basic auth, I assume you mean.

For http basic auth, you may as well just "secure" the resource by hosting it at http://my-username-and-password.example.com or http://example.com/username/password/resource

I thought https was implied these days :)

Yes absolutely https.

I wonder if someone from the Apigee team wrote these, as Google recently acquired Apigee[1], and the guidelines are mostly inline with what Apigee recommends.[2]

[1]https://techcrunch.com/2016/09/08/google-will-acquire-apigee...

[2]https://apigee.com/about/resources/ebooks/web-api-design

This guide has been in use since 2014, including recently launched Cloud Spanner API.

Disclaimer: co-author of the design guide.

And one more thing: Apigee's (awesome) e-book is designed to help customers write APIs.

The Google Style guide is a guide we use (and have been using for years) when designing our own APIs.

(I work on the API team at Google).

I don't see the actual guidelines, only Contents, Introduction and Conventions. On iOS Chrome /Safari. Also the fixed buttons overflows.

The page's responsive design is buggy. On narrow screens, the entire left column disappears. All the important content is only accessible from that left column.

I don't think it disappears, it just moves into the menu button, which seems fairly typical for navigation sidebars on narrow screens.

Do any of these API guides have good guidance around batch endpoints like handling a PATCH on multiple resources as a single request?

Not sure how common this pattern is, but one way I've seen this handled is to do a query to get a list of results, the document listing the results (or more commonly, the first page of results) has a link to a temporary resource representing the set of all results, which you can send a DELETE to remove all the members of the set. By analogy, you could send a PATCH to modify all the members instead.

This guide talks about batch get which is www.example.com/foos/123/bars:batchGet You could do something similar for other batches

I've used two strategies:

1. If it's a transaction (all-or-nothing) I POST a new transaction resource which references all the target resources. Remember that you can create as many resources as you want. There is no need to have a 1:1 mapping between your resources and the database.

2. If it's a batch statement (some-can-fail-some-can-lose) I simply stick to issuing multiple requests. This frees me and my clients from having to write complicated code that deal with partial success - and it can still be very fast especially with request pipelining.

If issuing multiple statements is too slow then I would attempt to increase the speed of the stack before adding the complexity of having to deal with partial success.

So in conclusion... No really good ideas on how to write batch statements. I don't really think REST APIs with their one HTTP response code maps very well for that use case. But I hope one of the two strategies above will be useful to you.

on a related note, anyone know a good saas for api documentation? preferably one that could take jsdoc imports or other code based generated docs...

I'm not sure exactly what you are looking for but try this tool from the swagger people:

http://editor.swagger.io/#/

It gets a little laggy for large swagger docs but its' quick and easy for smaller API docs.

Protocol Buffers...GraphQL...JSON-API...so many damn choices for API implementation! Next we need someone's essay of a blog post comparing/contrasting them all.

Also, the Protocol Buffers link in the 3rd paragraph is 404.

Thanks for catching that! Fixing it (it should have pointed to "proto3", not "proto3.md").

If you're using a good framework like C# Web Api, you don't have to choose. Just implement them all by adding them to the pipeline and the framework will automatically serialize/deserialize based on the accept header.

Interesting, do you have more details about this?

You register your custom serializer at application startup...

http://www.strathweb.com/2014/11/formatters-asp-net-mvc-6/

Your controller action looks like this:

public List<Employee> Get() { ..... return employeeList; }

Based on the serializers you have registered and the request Accept header, WebApi will serialize the list into JSON, XML, BSon (all built in) or a custom serializer that you add like Protocal Buffers

http://www.infoworld.com/article/2982579/application-archite...

Basically you can have all four registered and the client decides what it accepts.

Reply on news.ycombinator.com