How to Design Better APIs
r.bluethl.net
r.bluethl.net
The example uses a different code per issue, for instance: "user/email_required". Most integrators will build their UI to highlight the input fields that contain an error. Making them parse the `code` field (or special-case each possible code) is pretty toilsome.
// from blog post
{
"code": "user/email_required",
"message": "The parameter [email] is required."
}
Make it parseable: // improved
{
"message": "An email is required.",
"error": "missing_parameter",
"parameter": "user.email"
}
In addition, I:* rewrote `message` to be an acceptable error message displayed to (non-technical) end-users
* moved message to be the first field: some developer tools will truncate the JSON response body when presenting it in the stack trace.
---
As an added bonus, structured data allows you to analyze `error` frequencies and improve frontend validation or write better error messages: https://twitter.com/VicVijayakumar/status/149509216182142976...
I’d love to hear from anyone who has encountered APIs in the wild that actually implement this standard!
https://docs.microsoft.com/en-us/aspnet/core/web-api/handle-...
https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnet...
One thing to keep in mind re. "maybe-json-maybe-plaintext-maybe-empty" responses is that the more complex your errors, the more likely the error handling encounters an error. If you're trying to send back some JSON or XML but it fails it's usually better to at least shove out a line of plain text and hope it reaches a human than to mask the real error with a second fallback in the "right format" but with unhelpful fixed content.
You need codes because the field isn't going to be 'email' for much longer than it takes for your management to realize that people outside of the US also have wallets.
(While the comment I replied to can be read differently, I assume we all know that changing actual field names (in APIs) depending on localization is nuts)
Is that what you're saying? Or did I misunderstand you?
For more complex use cases, where we would want an error message to indicate that field value was too long and in addition provide maximum field length, we would need to introduce new field in the error response.
While it is solvable by adding this information to client application side. It would create a situation where the logic is duplicated in two places (backend and client application)...
Also if we would want better UX, then we would need to display all errors at the same time in the form that is incorrectly filled. This would require changing error structure to return array of errors and it potentially create a breaking change in the API or would result in confusing structure that supports both, legacy and new format...
Some years ago, I wrote an article sharing the ideas on how REST API error structuring could be done depending on the complexity of application or service: https://link.medium.com/ObW78jhDkob
Most validation I’ve seen looks like:
raise error if foo
raise other_error if bar
This pattern turns into one exception per response, and some foresight in architecting exceptions would be neededIf you need to validate some business logic like if sender's account has funds and receiver's account is not blocked, then this approach starts to look a bit strange. I would guess that the most of the time developers would implement checks so they would fail on first condition and would not check later one. This would result with single error in the array that kinda would look strange.
Validating business logic and returning several errors at the time requires good knowledge of the domain and in depth design of the system you are working. As this creates more complexity and slows down delivery, most of the time it is ditched and used only for particular use cases.
We could say that this would be more applicable to corporate solutions, but IMHO it really depends of the scale of the project, man power and the user experience you would want to create.
(Scroll to the very bottom)
Sure, there is gRPC, but it requires another API specification (the proto files).
There I said it. HTTP Verbs constrained REST APIS are the worst thing ever. I hate them.
They introduce un-necessary complexity, un-necessary granularity and they almost always stray away from the "REST principles". To hell with "Hypermedia" stuff.
I find it such a joy to program in server rendered pages. No cognitive overhead of thinking in "REST".
But, of course, all this is only where the client and server are developed by the same person / company.
For publishing data and creating API for third party use, we have no serious, better alternative to REST.
Is it in active use? Wikipedia page says last spec update was in 2010. No other details online. I could not find any specific implementations.
If you want a more advanced implementation with type inference for Typescript you can use this: https://github.com/shekohex/jsonrpc-ts
I'd still recommend implementing your own, my own implementation is based off the one above.
The only caveat compared to GRPC is you lose out on field validation due to it not being protobuf/and the obvious json decode overhead
Edit: if you want to know some current users of it, I believe the Ethereum protocol uses it heavily
I believe most users make use of it via an open source json-rpc python lib. You can find a few examples online if you'd like to know more.
In particular:
> The central idea behind HATEOAS is that RESTful servers and clients shouldn’t rely on a hardcoded interface (that they agreed upon through a separate channel). Instead, the server is supposed to send the set of URIs representing possible state transitions with each response, from which the client can select the one it wants to transition to. This is exactly how web browsers work
[1] https://www.oreilly.com/library/view/restful-web-apis/978144...
HATEOAS is a generalization of how something like a website would let a client navigate resources (through hyperlinks). It requires an intelligent agent (the user) to make sense. Without HATEOAS, according to Roy Fielding, it's not real REST. Some poor misguided API designers thought this meant they should add URL indirections to their JSON responses, making everything more painful to use for those unintelligent clients (the code that is consuming the API). Don't do this.
If you must do REST at all - which should be up for debate - you should keep it simple and pragmatic. Your users will not applaud you for exhausting HTTP verbs and status codes. The designers of HTTP did not think of your API. You will likely end up adding extra information in the response body, which means I end up with two levels (status code and response) of matching your response to whatever I need to do.
If something doesn't quite fit and it looks ugly or out-of-place, that's normal, because REST wasn't conceived with APIs in mind. Don't go down the rabbit hole of attempting to do "real REST". There is no pot of gold waiting for you, just pointless debates and annoyed users.
The best APIs I've ever used or built have been, at best, REST-ish.
And generally the parts where they deviate from REST make them more usable, not less.
I don't remember which API was that, I'll update the comment if I do.
Instead of building the logic to determine if, say, a Payment can be cancelled, based on its attributes, you simply check 'is the cancel link there'.
I find this a critical feature. Because between the backend, and various mobile clients, react, some admin, and several versions thereof, clients will implement such businesslogic wrong. Much better to make the backend responsible for communicating abilities. Because that backend has to do this anyway already.
I pointed him at HATEOAS and suggested if he wasn't familiar with it, he probably hasn't ever seen a truly RESTful API.
I don't think I convinced him that our approach is good (I'm not sure I am convinced either, but it works well enough for our purposes)
I do think I convinced him that "it doesn't correctly conform to a standard" isn't necessarily a useful critique , though. So that's a win.
What is the point of having your client translate an action to some operation on a document (read or write), only to then have your server try to infer what action was intended by said document operation.
It pains me that this article doesn’t mention any of the trade offs of each suggestion (POST vs PUT vs PATCH and expandable objects, especially) or of using REST APIs generally.
I bit my fingers every time I have mapped HTTP verbs and neither return codes, to REST verbs and codes.
Also, error codes at the API level often need a remapping when used in user context, for example if the OAuth token expires, we don’t say it the same way for an action of the user (then it’s mandatory) than when displaying data passively (in which case it shouldn’t be too red because the user may not care).
The basic reason you need binary identicality is the problem that a parser gives you an error state, by foregoing a parser you lose the ability to detect errors. And like you think you have the ability to detect those errors because you both depend on a shared library or something, and then you get hit by it anyway because you both depend on different versions of that shared library to interpret the thing. So you implement a version string or something, and that turns out to not play well with rollbacks, so the first time you roll back everything breaks... You finally solve this problem, then someone finds a way to route a Foo object to the Bar service via the Baz service, which (because Baz doesn't parse it) downgrades the version number but does not change the rest of the blob, due to library mismatches... Turns out when they do this they can get RCE in Bar service. There's just a lot of side cases. If you're not a fan of Whack-a-Mole it becomes easier to bundle all your services into one binary plus a flag, “I should operate as a Bar service,” to solve these problems once and for all.
I think you're misinterpreting "data transfer is in binary" with something like "a raw memory dump of an object in your program, without any serialisation or parsing step".
> but it requires another API specification
This implies API-specs are part of the problem; and I think they are not.
Specs that have generators for client-libs (and sometimes even sever-stubs) are verrrrry important. They allow us to get some form of type-safety over the API barrier which greatly reduces bugs.
One big reason for me to go with REST is OpenAPIv3: it allows me to completely spec my API and generate clients-libs for sooo many languages, and server-stub for sooo many BE frameworks. This, to me, may weight up to the downsides of REST.
GraphQL is also picking up steam and has these generators.
JSON-RPC (while great in terms of less-overkill-than-REST) does not have so much of this.
A better perspective would be, most softwares can be viewed as a set of domain specific objects and the set of operations (verbs) that can happen to those objects. These operations may not be a single attribute update, but a more complex dynamic set of updates over a variety of business objects. If you try to model this with a REST api, it either quickly becomes chatty or you end up compromising on REST principles.
GraphQL seems to make much more sense than REST, IMO.
If you have to check the API docs anyways, I rather define custom domain specific verbs than debate whether something should be a PUT or a PATCH.
http://harmful.cat-v.org/software/xml/soap/simple
I was puzzled, at the time, why the industry was cluttering HTTP in this way. Why not establish a clean protocol for this?
But people kept getting distracted by something that seemed like maybe it would solve the problem.
Dave Winer used to be a very big deal, having created crucial technologies for Apple back in the 1980s and 1990s, and he was initially horrified by JSON. This post is somewhat infamous:
http://scripting.com/2006/12/20.html#godBlessTheReinventers
"... and damn, IT'S NOT EVEN XML!"
He was very angry that anyone would try to introduce a new serialization language, other than XML.
My point is, the need for a clear a clean RPG protocol, but the industry has failed, again and again, to figure out how to do this. Over and over again, when the industry gets serious about it, they come up with something too complex and too burdensome.
Partly, the goal was often too ambitious. In particular, the idea of having a universal process for serializing an object, and then deserializing it in any language, so you can serialize an object in C# and then deserialize it in Java and the whole process is invisible to you because it happens automatically -- this turned out to be beyond the ability of the tech industry, partly because the major tech players didn't want to cooperate, but also because it is a very difficult problem.
It all came to a head when someone wrote an endpoint using a PATCH verb that some people were adamant should have been a PUT.
It was among the most silly nonsense I have ever been a part of and these discussions have thankfully gone to zero since we decided on only GET and POST
> 15. Allow expanding resources
I would suggest the opposite. A REST API should not return nested resources at all. Instead, and to stay with the example provided on the website, to obtain the "orders", the /users/:id/orders endpoint should be called.
It might be tempting to return nested resources, because clients would only have to make a single call.Technically this is true but once the domain of your API starts to grow, you will find that the interface will become increasingly muddled.
The suggestion provided (use query parameters) is basically a work-around. If you want to offer this to your clients, front your REST API with a GraphQL API instead. It is literally the problem that GraphQL solves. Keep your REST API clean and dumb and focused around CRUD.
Anytime clients need to make 15 async calls before the UI can be displayed, you're headed up the creek. Generally speaking, this is an anti-pattern. There are exceptions, but they're not the rule.
It's better to weigh the tradeoffs in any given situation and make a decision about bundling sub-resource references based on what you're optimizing for.
A few quick examples:
* Dev speed: Unbundled
* Quick-loading UI: Bundled
* Sub-resources are computationally-expensive to query? Unbundled
This has been my experience consistently throughout 20 years of web-tier development.
The true power with GraphQL is that it might not even make all 15 calls because it will entirely depend on what you are querying for. E.g. if you query for a User but not the Orders for that User, then the request to retrieve the orders is simply skipped by GraphQL.
I have only used Apollo for GraphQL, and I found a few things about it offputting (e.g. I need a 3rd-party library to figure out what fields the client actually requested). What GraphQL server do you use? Or is Apollo + Express generally a good "default" option for basic setups?
The comment you're replying to set out specific reasons why they disagree with expanding/bundling sub-resources. It obviously depends on your use case - and in fact they say use GraphQL, which I heartily agree with - but the point is that you don't always know how the API is going to evolve over time, and keeping things unbundled tends to be a "no regrets" path, while bundling resources by default, in my experience, can lead to trouble later.
When the API evolves - as it probably will - using bundled resources ends up running the risk of either an inconsistent API (where some stuff is bundled and some isn't, and the consumer has to work out which), a slow API (because over time the bundled data becomes a performance burden), or a complicated API (where you need to embed multiple, backward-compatible options for bundling and/or pagination in a single operation). In addition, the bundling of resources commits you to I/O and backend resource consumption based only on an assumption that the data is required. None of this makes sense to me.
In practice, if you can keep your API latency reasonably low and take a little bit of care on the client side, there's no reason a user should notice the few milliseconds of additional latency caused by a couple more API calls during the page draw of an app.
It's not about architectural purity, it's about decomposing your application in a way that balances multiple conflicting needs. I agree with pan69, after many years of doing this in multiple contexts, my default has become to not bundle resources when responding to an API request.
One thing to add, is that there's nothing preventing you to invent a new noun for a particular rest resource that returns bundles of content. eg. /user/:id/dashboard - it makes it so this endpoint is not tied to eg. user/:id .. making that endpoint harder to change in the future, but also solves the issue mentioned by the rude comment above re: needing to perform a lot of separate rest calls.
Agreed even at rate limits comment - frequently, probably in moments of desperation, they rate limit according to Host header to keep their own products up and working.
Can you elaborate what's terrible about the developer experience? If anything it's much better than REST, even if the developer of the API doesn't bother with documentation, the GraphQL schema is fully usable as documentation. Plus the way to input and output data into the API is standardized and the same across all GraphQL API's.
> terrible performance
How? Is the API slow to respond? Or are you making too many calls (which you shouldn't do) increasing the round trip time?
> terrible uptimes
I fail to see how that uptime is related to GraphQL. REST API's can be just as unreliable, if the server is down, the server is down and no technology can fix that.
You want
GET /orders?user=Id
Orders can be searched on many dimensions, not inherent to users
But then again, when it comes to API's, domain modelling is the hard part and it is therefore the reason why you don't want to return nested results/objects.
Will I? For example The Stripe API uses expand parameters and I prefer that approach to "atomic" REST or being forced to use GraphQL. There is a missed standardization opportunity for incremental graph APIs built on REST.
Ok, but an order has an array of order lines, and each order line has sub-arrays as well, like details of the packages it was shipped in etc.
So that might be 5-10 calls to get an order line, and for a few thousand lines we're looking at several tens of thousand calls to get a full order.
Secondly, you then make some changes and want to replace the order with the data you got. You then have to produce a delta and upload that. And those thousands of calls better be done in a transaction of sorts, otherwise you'll have big issues.
Seems easier to me to just be able to GET or PUT an entire order with all the details in one go.
> So that might be 5-10 calls to get an order line, and for a few thousand lines we're looking at several tens of thousand calls to get a full order.
You definitely need to pick and choose the granularity you offer. There's a level of normalization in data structuring that approaches absurdity and this would be a good example of an absurd case.
It is however an excellent demonstration of why making overbroad "you must never do X" rules is dangerous.
I think a decent test is to ask yourself the question: "How much data is here that I don't need?"
In the example referenced, if I'm looking at the full order history, I probably want to see only summary/header data with the option to view the full order details, so it wouldn't make sense to return to the user potentially a hundred thousand order lines' worth of data just because they're trying to view their history.
If I then want to view the /orders/:id for one specific order, at that point it does make sense to return all of the related lines, shipment details, etc.
A nested resource is hard to optimize. Simple, atomic, flat resources can be cached (by clients, proxies or server) much more efficient. Once you allow nested resources, there's no going back, as clients will depend on them. So you've effectively disabled lots of performance improvement options.
A nested resource implies data relations. Tightly coupled to a data model. One that will change over time, yet the api is hard to change. If you have a Project nested in your Users, and the business now needs multiple projects per user, this is hard to change with nested resources, but much easier with endpoints, the /users/:id/project can be kept and return e.g. the first project, next to a new /users/:id/projects.
To take that User/Project example: the Project and User have different transaction boundaries: it matters not that a User.username is updated while a Project.projectname is updated differently. But it does matter when a username and password get updated simultanous. And -for the sake of the xample- it matters that the user + users-roles must be within one transaction boundary.
By allowing nested resources on mutations you essentially break the bounded contexts and the transaction boundaries.
By not allowing nested resources on mutations, but allowing them on-read, you introduce inconsistencies: the resources on-insert differ from the resources on-read. This becomes even more problematic when you do allow certain nested resources on mutations but not on others (because, behind the scenes you've determined a bounded context/transaction boundary). To an API consumer this is entirely arbitrary. Why can I "PUT user.roles[1]" but not "PUT user.projects[1]", yet have both included at wish on-read?
With nested resources, here too, you paint yourself in corners. They take away a lot of choices you likely want to make in future. Being a bit more restraining in what you allow clients to do, keeps those options open for you. In this case, keeps the option to move transaction boundaries when business needs require (and they will).
Since this field is validated in the DB, merely having `v1` `v2` distinction at the API layer is not sufficient. So I was thinking we will have to either 1) disable DB validations and rely on app validations or 2) run two separate systems (e.g., one DB per version) and let people 'upgrade' to the new version (once you upgrade you cannot go back).
Even though people refer to Stripe's API versioning blog, I don't recall any mention of actual data model changes and how it is actually managed
For other fields like your example of `avatar_url` I would use a placeholder avatar for all legacy users (the grey anonymous snowman profile comes to mind).
For example, in financial services and insurance, regs change and what data we need to collect change and sometimes their dependency will change. I am curious what's companies that have grown substantially had to do to their APIs.
We're working in this space at the moment, (eliminating the pain from breaking changes in APIs) and looking to get feedback on what we're building.
We're all from banking backgrounds, so understand the reg headaches you're talking about.
Can we chat?
Coming at this from a strongly-typed background, you acknowledge the fact that despite new regulations requiring a scan of the user's birth certificate in order to get an API token, that field can't be marked as non-null if you don't in fact have all those birth certificates. You are then forced to handle both the null and not-null cases when retrieving the value from the database.
So your API v2 can absolutely (in its MVC or whatever model) have that field marked as non-null but since your API v1 will still be proxying code to the same database, your db model would have that field marked as nullable (until the day when you have collected that field for all your customers).
If a downstream operation is contingent on the field being non-null, you are forced to grapple with the reality that you don't have said field for all your users (because of APIv1 users) and so you need to throw some sort of 400 Bad Request or similar error because (due to regulations) this operation is no longer allowed past some sunset date for users that haven't complied with regulation XYZ. In this case, it's a benefit that your db model has the field marked as null because it forces you to handle the cases where you don't have that field.
I guess what I'm saying is the db model isn't what you wish your data were like but rather what your data actually is, whether you like it or not.
Obviously there will be always challenges with eventually consistency but that is another topic altogether.
It uses Accept and Content-Type to version resources: application/vnd.company.article-v1+json
I only half agree with the sentiment that /api/v1 violates REST patterns. I don't think there's any guarantee that /api/v1/bars/123 can't be the same object as /api/v2/bars/123.
Then if we absolutely do need a field non-null in the db, we run a backfill with either derived or dummy data. Then we can make the column non-null.
We use alembic to manage migration versions.
But we aren't even out of v0 endpoint and our stack is small enough that we have a lot of wiggle room. No idea how scalable this approach is.
The downside is maintaining separate api and db models, but the upside is decoupling things that really aren't the same. We tried an ORM which has a single model for both (Ormar) and it just wasn't mature, but also explicit conversions from wire format to db format are nice.
https://test.salesforce.com/services/Soap/c/47.0
And in the latest version of the API docs they have details regarding old versions. Example:
https://developer.salesforce.com/docs/atlas.en-us.api.meta/a...
Type reference Properties Create, Filter, Group, Nillable, Sort Description The ID of the parent object record that relates to this action plan.
For API version 48 and later, supported parent objects are Account, AssetsAndLiabilities, BusinessMilestone, Campaign, Card, Case, Claim, Contact, Contract, Financial Account, Financial Goal, Financial Holding, InsurancePolicy, InsurancePolicyCoverage, Lead, Opportunity, PersonLifeEvent, ResidentialLoanApplication, and Visit as well as custom objects with activities enabled.
For API version 47 and later, supported parent objects are Account, BusinessMilestone, Campaign, Case, Claim, Contact, Contract, InsurancePolicy, InsurancePolicyCoverage, Lead, Opportunity, PersonLifeEvent, and Visit as well as custom objects with activities enabled.
For API version 46 and later, supported parent objects are Account, Campaign, Case, Contact, Contract, Lead, and Opportunity as well as custom objects with activities enabled.
For API version 45 and earlier: the only supported parent object is Account.
b) Pagination is the devil. The state of the server can change while the user is paginating, leading to fragile clients. Don't paginate your API. If you think you have a need to paginate, have one API call return a list of IDs, and have a separate API call return a list of resources for a given list of IDs, where the second API call accepts some maximum number of IDs as a query parameter. This ensures consistency from one API call to the next.
If I do a search, a paginated search result can have an ID so I can paginate between the data without a new data messing up my pagination.
But for normal entities, simple pagination is (mostly) more than enough.
The solution you are describing is overkill and almost no benefit at all.
The problem is not pagination, and the solution is not to avoid pagination. The problem is offset-based pagination, and the solution is to use cursor-based pagination.
> If you think you have a need to paginate, have one API call return a list of IDs, and have a separate API call return a list of resources for a given list of IDs, where the second API call accepts some maximum number of IDs as a query parameter.
This has the same problem as you described above. The state of the server can change between fetching the list of IDs and operating on them.
> The state of the server can change between fetching the list of IDs and operating on them.
Right, for example, a returned ID may have been deleted before the user can issue a query for that resource. But this is usually far more comprehensible to clients, particularly if the resource requires authorization such that only the client is permitted to delete that resource.
It is quite simple to implement pagination in the application layer using a unique record identifier (primary key or ULID or ...) as an anchor for the navigation. From that unique ID, we can then fetch the previous `n` or the next `n` records, depending on the direction of the navigation.
This way, the server remains stateless, since the anchor (possibly sent as an encoded / obfuscated token, which can include some other parameters such as page size) is supplied by the client with each pagination request.
Unless I am missing something in your argument.
Anyway, there is plenty of documentation out there for cursor-based pagination; Hacker News comments isn’t the right place to explain the implementation details.
2. The WHERE condition typically uses `>` or `<`, so it doesn't fail even when that record is deleted before the next client request referring to it.
Cursors take too many server resources and require client affinity.
I think pagination is only predictable under these conditions:
1) The offset used for the next fetch must be based on a pointer to a unique key. We can't rely on the number of rows previously seen.
With this rule, deletes which occur to rows in previous pages will not cause unpredictable contractions.
2) The paged result set must be sorted based on a monotonically increasing field, like created_at, plus enough other fields to create a unique key. You could lean on the PK for this, i.e.: ORDER BY (created_at, id) ASC.
With this rule, new inserts which occur during page enumeration will only affect unseen pages (and we'll see them eventually)
The API call looks roughly like this:
/orders/?region=US&offset=(2022-03-12T07:05:58Z&ord_1234)&limit=100
The DB query looks roughly like this: SELECT *
FROM orders
WHERE (created_at, id) > (:offset_created_at, :offset_id)
OR (
:offset_created_at IS NULL
AND :offset_id IS NULL
)
ORDER BY (created_at, id) ASC
LIMIT :page_size
EDIT: formattingThe solution is usually simple: assume it doesn't matter. Write a spec for your feature and explicitly state that "In the solution we assume it doesn't matter if the same record is ocassionally reported twice or a record is missing from the pagination in some cases". Done.
- Always use a response envelope
- HTTP status is not the same as your application status. I always include a "status" field in the response envelope (OP recommends standardizing errors but I think standardize all of it)
- Always have an unique error code for every error your API can return (they should also be URL safe ("some-thing-went-wrong"), basically an enum of these should exist somewhere, the more specific the better.
- Offer OpenAPI whenever you can.
- There is no such thing as a publicly exposed "private" API. If it can be hit (and is not protected), it eventually will be.
- Do blackbox testing of your API, E2E tests are the most important kind of test you could have.
- (controversial) build in special test/debug endpoints -- these help with blackbox testing.
Edit:
Nevermind, see [0]. It's the simple and intuitive concept of encasing the payload in an consistently structured object which includes metadata, e.g.
{"Error": null, Data: ...}
[0] https://stackoverflow.com/questions/9989135/when-in-my-rest-...Makes intuitive sense, as it's then easier to develop a nice, generic REST client.
export enum ErrorCode {
InvalidEntity = 1,
NotSupported = 2,
UnexpectedServerError = 3,
InvalidRequest = 4,
Validation = 5,
// ...
}
export enum ResponseStatus {
Success = "success",
Error = "error",
}
export class ResponseEnvelope<T> {
public readonly status: ResponseStatus = ResponseStatus.Success;
public readonly data: T | Pruned<T> | null = null;
public readonly error?: {
code: ErrorCode,
message: string,
details: object | string,
status: ResponseStatus,
};
constructor(opts?: Partial<ResponseEnvelope<T>>) {
if (opts) {
if (opts.status) { this.status = opts.status; }
if (opts.data) { this.data = opts.data; }
if (opts.error) { this.error = opts.error; }
}
if (!this.data) { return; }
// Prune if the thing is prunable
this.data = prune(this.data);
}
}Hopefully this isn't too hard to grok, it's Typescript.
BTW, If you're cringing at the number scheme for the ErrorCode enum, don't worry I am too. This is why I prefer to use strings rather than numbers, and it basically just ends up being stuff like "invalid-entity", etc.
I find envelopes to be unnecessary cruft that just junk up otherwise clear code. And I think packing metadata into an envelope along with the actual data keeps people from thinking clearly about their own APIs, mostly with respect to ambiguity surrounding the word “error”. Validation errors are errors and network failures are errors, but they’re very different animals and should never be conflated.
I don’t want to check for a status code in metadata ever, when an HTTP status is provided. I don’t want to read a time stamp if that time stamp is simply a property of the request. However if the time stamp is a property of the data, then I care - but then it shouldn’t be part of the metadata.
> Validation errors are errors and network failures are errors, but they’re very different animals and should never be conflated.
> I don’t want to check for a status code in metadata ever, when an HTTP status is provided.
These seem like conflicting views. If the HTTP status is all you check you must be conflating application/operation status and network status.
> I don’t want to read a time stamp if that time stamp is simply a property of the request. However if the time stamp is a property of the data, then I care - but then it shouldn’t be part of the metadata.
Sure -- I'd like to say that I think the disagreement here boils down to this question: should metadata be represented in data responses.
Correct me if I'm reading you incorrectly, but it seems like you're arguing that metadata should not be represented in the response, only data. I'd argue that the in-practice and academic standards dictate the opposite, with standards like hypermedia, json schema, HAL, JSON:API, OpenAPI, etc.
If it's just a question about the degree of metadata that should be present then that's one thing, but it sounds like you're against it in general. Once you have any metadata you must have an envelope of some sort, by definition (whether it's a good or bad one), as far as I can see.
{"data": <base64_encoded_video>, "status": "whatever"}
Of course not!You already have an envelope, it's the HTTP protocol. The real trick is generically mapping data structures to HTTP responses with headers. In fact HTTP-response-shaped-objects make halfway decent general purpose envelopes in the business logic itself. They are basically Result[T, E] but with even more metadata available.
And no, we don't do that for static content for a simple reason: static content isn't served from our API servers.
I would mostly agree except in 1 case, streaming data is easier without an envelope. Making some array inside an envelope stream is usually more code and messier than just getting some metadata out of header. So if you have something like data integration endpoints and you expect someone could pull many megs of records, consider no envelope.
I agree, and would go so far as to say streaming data would probably have a different delivery mechanism all together - SSE/websockets/etc. if you were doing long polling I may still want to put the metadata in the body but I agree it could be kept out of sight quite nicely in the headers.
An anti pattern I’ve often seen has devs avoiding 5xx errors in bizarre ways. I would change the above to make to have monitoring in place to address 5xx errors. By all means, let your code throw a 500 if things go off the rails.
Generally agree with everything else but strongly disagree with pagination envelopes and offset pagination. Allow keyset pagination using ordering and limiting query parameters.
I’ve been designing SDKs for a long time; APIs, for a somewhat shorter time.
I don’t like “pure” RESTful APIs, because I feel as if they are “human-hostile.” I tend to follow the patterns set by companies like Google, where the stimulus is a fairly straightforward URI (with parameters, as opposed to a block of XML or JSON —if possible, as I know that we often still need to send big data as transaction data, which can be a pain, depending on the method), and the response is a block of structured data. That also makes it a lot easier for API clients to build requests, and allows the server to “genericize” the processing of requests.
> 10. Use standardized error responses
Is something I do, along with a text header payload, describing internal information about the error. It is my experience that this text payload is never transferred, and I am forced to send a text response via the body, if I want to know internal information. That can be a pain, as it often breaks the transaction, so I have to play with the server and client, frequently, using a cloned server instance.
I should note that my forte is not backend development, so the chances are good that I am not familiar with all the tools at my disposal.
Can you elaborate on this? Give examples?
I should note that the BASALT layer uses “plugins,” that can be extended to include “pure” REST APIs. I just haven’t found these practical, for my purposes.
[0] https://riftvalleysoftware.com/work/open-source-projects/#ba...
Many rest apis are lazy and developer friendly, not consumer friendly.
If you have related resources, let’s say, product and product options as two distinct endpoints:
- /api/product
- /api/options
Then, and I want to be clear here, it is impossible for a client to perform an atomic operation on multiple distinct objects types.
Let’s say the client needs to add a product with a single option or fail.
You can create a product.
You can create an option.
You can add an option to a product.
…but, at some point in time, a product will exist with no options on it.
This kind of “pure” rest api is simply convenient to the developer because they push the problem of object consistency to the client.
…but that’s not a client concern.
If a product needs to be associated with an option on creation, then your api should offer that as an endpoint.
It doesn’t meet the “standard” for a rest api?
Too bad.
Write APIs that do what the customer / consumer needs not lazy APIs that just make your life easier.
I’ve worked with a lot of APIs and you know what I do not give the tiniest moment of care about?
Consistency.
I do. Not. Care. If POST /api/foo creates an object, or if the endpoint is /api/foo/create.
Messy? Inconsistent?
I don’t care. Just put it in the documentation and I’ll find it and use it.
…but if your api forces object relational consistency onto me as an api consumer, you have no idea how much pain and hassle you’ve caused me having to implement my own set of fake transactions over your stupid api.
Please, write APIs for consumers, not according to “the rules” of rest APIs.
…and provide documentation. :)
"congratulations, we've provisioned the bundle A, C, C, C that you ordered! sorry about B!"
A good example is the tip on PUT vs PATCH to update objects. That seems to be missing the point. Why are you forcing the clients to calculate the correct resource fields to PATCH to the server? This is supposed to be an API, not a database. Just expose methods that correspond to the actions that the users will perform.
Sure, HTTP only has 5 verbs, but that doesn't mean your API should solely consist of five possible interactions on a resource.
More like 40.
You can also just like...make up your own verbs, the HTTP police won't arrest you, though this rustles the jimmies of HTTP purists. No guarantees any middleware will support it, and this is probably a bad idea if you actually do this in public apis, it's more so "hey, these verbs aren't magic incantations, it's just a string which gets switch-cased".
Same with HTTP status codes. There's obviously some well known ones but you can just like...make up your own.
https://www.iana.org/assignments/http-methods/http-methods.x...
but, if you've already suffered through the pain of attempting to implement some kind of best-effort ad-hoc distributed commit logic in your client code, then needing to call two completely different services for api/product and api/options doesn't really make anything _worse_.
on another hand, it doesn't make anything better.
Outside of the bigger providers I don't think most APIs are designed with general consumption in mind. The impetus usually seems to be reactive - "we've got this idea for a mobile app using data from our legacy WMS/CRM/whatever, and it needs to be finished yesterday!"
So generally (not always but usually) what I've seen is whatever the underlying system supports and requires gets reflected into the API, and there's minimal proper engineering or sensible abstraction, partly because of time limits and also because the requirements are discovered mid-flight.
"Oh... Products have options in a separate entity? Oops. Uh, we'll add an options endpoint. Easy."
Several years later, the app is mildly successful, the business decides to whitelabel the API for some of their B2B clients, there's no budget for a redesign, "It works doesn't it? If it ain't broke don't fix it..."
Where possible I try to design and build APIs with some forethought and push for as much information on requirements and capabilities of existing systems beforehand but quite often the stakeholders don't really know and don't want to pay for a few weeks of in-depth analysis.
The endless facepalming of hearing the cycle of "Don't overengineer it" when I ask too many questions people can't answer, through to "Why didn't we pick up on this requirement" and having to listen to "we can all learn from this" when something fails because of some if/else statement buried in the legacy business logic, right back to "Don't overengineer it" when you propose the clean way of encapsulating that in the API, has left a permanent indent on my skull. So much leakage of black-box logic into the clients which bloats frontend logic and provides plenty of double handling and scope for unknown business logic to be missed, and no one in charge really gives a shit other than the developers and engineers, and that's fine because we can deal with it and it's their fault for not thinking of it anyway... they're the smart ones right?
You'd think this is just a problem with SMEs but some of the biggest national and multinationals I've worked with have a ton of horrid shit in their API designs. One industry-wide government-mandated reporting spec you'd post a SOAP envelope that wraps an undocumented XML document (they had an example document and the envelope was documented, but no doctype or anything to go off for the contents), a major real estate interchange format has you polling a REST API for an update flag, then loading and saving CSVs over FTP. Some godawful affiliate marketing API essentially just exposed their nth normal form relational database as an API. One government department just used HTSQL for their API using a python script to reflect some of their silos into PGSQL and called it a day (even though whether or not your interactions would work or made sense depended entirely on their internal application logic which wasn't represented or documented).
And too many projects where it turns out the promised APIs didn't even exist until the app was well into development. "Don't worry about it, that's their problem and gives us an excuse when the inevitable delays occur..."
No wonder we're hearing of massive breaches daily, from top to bottom the entire industry is barely held together by glue and sticky tape. It seems like an engineering problem but I think it goes much higher than that, it's a cross-industry failure in planning and management. Engineers are not given the time and budget to do the job that the data deserves. The market self corrects though, massive breaches have closed more than one business down for good.
It feels futile but I do support and encourage people to work towards just reasonable and well-documented endpoints, I doubt there's one true standard, but just some level above slapping it together at the last minute would be nice. I don't care if it's JSON or XML, or whether paths follow some semantics or another. As long as the relationships, types, and mandatory vs optional fields are explained clearly and succinctly, and the auth mechanism isn't a nightmare, and I can work with it.
Sorry for the rant this is just one area that triggers me. I've seen too much bad practice, and fighting against it for decades and still hearing the same lack of concern in the architecture phase has left me more than a little jaded.
There is nothing nonstandard about such a REST api. If a product must be associated with an option then the /product/create endpoint should only accept a product with an option already attached.
> This kind of “pure” rest api is simply convenient to the developer because they push the problem of object consistency to the client.
It's not like REST means that to be "pure" or "standard" you end up exposing all your database tables with read/write and just let callers CRUD anything they want, including creating inconsistent data?
> “the rules” of rest APIs.
The rules are very simple: the REST api should expose and enforce the consistency rules of the business logic!
> It doesn’t meet the “standard” for a rest api? Too bad.
It is perfectly fine to model the creation of a product with options via a new resource. Another possibility is to model this as a transaction, a series of requests. Both meet the goals of atomicity and also following restful constraints.
But since you disallowed those possibilities as the premise, there is no constructive way forward from there. Nice construction of a straw-man, must feel satisfying to see it topple over, but who do you hope to impress with that?
A lot of implementation are actually based on an ISO draft which changed in final release. But most dev do not have access to the spec. For example, timezone can only be specified as offset in the standard, while implementations accept name.
Globally avoid ISO for software, it is non free crap.
Also, do not use those standard for dates with timezone in the future. Use wall time/date and location. As timezone can change way more often than you think.
Of course, when preparing queries against the API, you may need a helper to build valid timestamps. But this is mostly relevant to the discovery phase which isn't automated. And textual dates also have drawbacks, for instance the need to encode it when used a an URL parameter.
Example: in November you create an appointment in 6 months at 15h, but then your government decides, to not use summer time next year.
If you use timestamps your appointment will be wrong by one hour (humans tend to keep the 15h).
In general, I am a huge fan of timestamps, but I think it is good to know where they have their limits.
Why would you ever want anything to do with timezones in an API? Even for appointments, it's still a timestamp. Timezone should be applied the very last moment, as part of date formatting for output on the client.
If you allow your users to create appointments this much in advance and then they miss them, it's a UX problem, not an API design problem.
In the example in the post you’re replying to, the time zone change means the time stamp would be wrong by an hour. This doesn’t mean that people should show up an hour earlier or later than initially intended!
Timezone isn't a simple constant offset either. For places that use DST, any decent date library would take that into account. If you entered a date in summer while it's winter, it would still result in a correct unixtime.
It's a necessarily manual process and it creates a lot of confusion either way. And it's sufficiently rare to ignore. And these changes tend to be announced in advance.
And we've had examples in the past years of this happening with only weeks of notice, so no, it's not something that never happens.
But who cares. We told you, that there are valid use-cases where timestamps are not the best solution and brought examples. If you don't want to see them, it is your loss.
While pagination is important, there is another possibility of pure size - it is using cursors, like mentioned in the JSONAPI specification[1] (containing many of the hints in the topics' post) and in this blog post[1]
[1] https://jsonapi.org/format/#fetching-pagination
[2] https://dev.to/jackmarchant/offset-and-cursor-pagination-exp...
I really hate when APIs use different api-key headers depending on the role of the consumer.
It is very annoying when you get dates that are not in the ISO format. There are reasons to not use UTC everywhere. One should make that decision.
The reason why many APIs use POST instead of DELETE is that POST is said to be more secure.
Many APIs that I use do not have neither of PATCH, PUT or DELETE. An order for instance will have an order status resource that one just keeps adding status entities to. In general, well-designed systems minimize the need for changing data.
Wouldn't a bearer token [1] make more sense? Defined for use by OAuth2, but I don't see why it couldn't be the general mechanism for... bearer tokens.
> 11. Return created resources upon POST
Especially important if your database+caching layers use eventual consistency.
/books
/books/:id
/book
/book/:id
It gets harder to keep consistent when there are a lot of nouns that have the same singular/plural form like "clothing" /books
/book/:id /clothes/:clothing_id
The reason for using plural is because without the `:clothing_id`, `/clothes` endpoint would return a collection of clothing.It's stupid to have /geese return a Goose.
If the APIs were in Esperanto it would be different.
Helped me get started with API design in my early career. Learning from other existing APIs helps too. such as stripe, github, shopify. Any others?
Something that We do at my current job:
* set a standard and stick with it. tweak it if needed. we even have naming standard on some of the json key. for example, use XXX_count for counting. when it doesn't make sense, use total_XXX.
* Document your API, we use postman and code review API changes too.
Part of a good API design is documenting it. If you use an API specification format, such as OpenAPI, design and documentation overlap nicely.
POST /document/:id?params
Creates a document with parameters
This is one reason why I don't bother with these methods and stick to GET and POST.
Most people don't bother with them. If you need caching or want to be able to manipulate params in the browser/link to resources, use GET.
This idea that you need additional verbs for web services is a classic case of in-theory vs in-practice. Introducing non-trivial complication for very tiny benefit is a strange tradeoff.
` list of uuids -> list of results `
is usually bad design to begin with.
Past dates should always be in UTC, for the same reason - timezone rule changes are sometimes even retroactive.
2. The ISO date format is "big endian" by default, which makes it trivial to chunk and compare just specific date parts. Want all events that occurred on the same (UTC) date? Just do dateField.substring(0, 10). Everything in the same year? dateField.substring(0, 4).
Edit: a relevant link
A ven diagram very cool
* Supports birthdays for people older than 52
* Reliable after 2038
* Supports leap seconds
* HTML date/time spec is a subset
* String collation order matches temporal order
* Excel
Computers are the main consumers of APIs, and ISO 8601 is far from machine-readable.
For example, in Elixir, DateTime.from_iso8601/1 won't recognize "2022-03-12T07:36:08" even though it's valid. I had to rewrite a chunk of Python's radidjson wrapper to 1-9 digit fractional seconds (1).
I'm willing to bet 99% of ISO8601 will fail to handle all aspects of the spec. So when you say "ISO8601" what you're really saying is "our [probably undocumented, and possibly different depending on what system you're hitting] version of the ISO-86001 spec."
(1) https://github.com/python-rapidjson/python-rapidjson/pull/13...
No.
The specific profile of ISO8601 that should be used to express timestamps in an API is that defined in RFC3339.
Choosing to use a half-baked parser is a separate matter.
Using PUT to create elements means that the client has to supply the ID, but this is easily solved by using GUIDs as IDs. Most languages have a package for create a unique GUIDs.
In your example (and I ask this as I remained confused after also reading SO):
Let's say that you need the client to provide the ID in the request body.
In this case, how is using PUT when creating a new resource idempotent if the ID should be unique and you have a constraint on the DB level for example?
What happens when the second call will be made with the same ID?
If I execute the following sequence:
Call 1: PUT resource + request.body {id: 1} => a new record is created so the state of the system changes
Call 2: PUT resource + request.body {id: 1} => no record is created, maybe the existing one is updated
IMO this is not idempotent nor should it be. Creating a new resource is not an idempotent operation.
I also don't like that depending on the state of the DB two requests with the same attributes will have different impacts on the system.
In my mind as a consumer of an API it is simpler: POST it creates a new record so I know what to expect, PUT updates an existing one.
You say the same request has a different impact on the system, but what that means is the system converges to the requested state.
Maybe this is overkill if your retry strategy is to have a user in the loop, but I don't see how it's simpler for the client even without any retry behavior (if/else new/reuse uuid vs if/else put/post).
"From my experience, there barely exist any use cases in practice where a full update on a resource would make sense"
Just in finance I can think of *dozens* of use cases around invoicing and purchasing *alone*. A lot of times thousands of resources may need just one field "corrected", but hundreds others "reset" to ensure the proper retriggering of routing, workflows, and allocations. That the resources need to exist as originals is incredibly important for these kinds of things.
Nevertheless, as a (largely) non-web dev I found the article interesting and useful.
I kid. There’s some good stuff in here.
2. ISO 8601 is a shit show of a standard. Last time I surveyed, there was not a single compliant implementation in the world. Recommend "Date and Time on the Internet: Timestamps" <http://rfc-editor.org/rfc/rfc3339> instead. This standard is public/not encumbered by stupid copy fees and is restricted to a much more reasonable profile that can actually be implemented fully and in an interoperable fashion.
4. Use `OPTIONS ` instead. <http://rfc-editor.org/rfc/rfc7231#section-4.3.7>
5. This goes against the design of the Web. Do not version URIs. If the representation changes and is not backward compatible, change the media type instead, e.g. by adding a profile <http://rfc-editor.org/rfc/rfc6906>. You can serve multiple representations on the same URI and vary on the request headers (e.g. Accept-).
6. HTTP already contains a customisable mechanism. Use the standard header <http://rfc-editor.org/rfc/rfc7235#section-4.2>, not the custom header `Api-Key` which is not interoperable.
7. "Don't use too many [HTTP status codes]": why? This opinion is not backed up with an explanation. The correct advice is: use as many status codes as arise from the requirements. `422 Unprocessable Entity` is useful in nearly every Web system.
8. What does "reasonable" mean? This lacks an explanation.
10. Use application/problem+json <http://rfc-editor.org/rfc/rfc7807> instead.
11. "It's a good idea to return the created resource after creating it with a POST request": why? This opinion is not backed up with an explanation. If you follow this advice, the user agent cannot programmatically distinguish between a representation reporting on the requested action's status, and the representation of the newly created resource itself. Use the Content-Location header <http://rfc-editor.org/rfc/rfc7231#section-3.1.4.2> to make that possible, use the Prefer header <http://rfc-editor.org/rfc/rfc7240> to give the user agent some control over which representation to respond with.
12. "Prefer PATCH over PUT": disagree, ideally you offer both since there is a trade-off involved here. The downsides of PATCH are not mentioned: the method is not (required to be) idempotent meaning it becomes moderately tricky to keep track of state, and the client is required to implement some diff operation according to the semantics of the accepted media type in the Accept-Patch header which can be difficult to get right.
13. Missed opportunity to advertise the OPTIONS method and related Accept-Post <https://datatracker.ietf.org/doc/html/draft-wilde-accept-pos...> / Accept-Patch <http://rfc-editor.org/rfc/rfc5789#section-3.1> headers. A representation of a specific media type can self describe with the "type" Link relation <http://rfc-editor.org/rfc/rfc6903.html#section-6>.
14. Don't mix data with metadata. Use the Range header <http://rfc-editor.org/rfc/rfc7233> and next/prev Link relations <https://webconcepts.info/concepts/link-relation/next> instead.
15. Instead of complicating both server and client, the better idea is to simply link to related resources. We are not in the 1990s any more. A well written Web server offers HTTP persistent connections, HTTP pipelining, all of which make round-trips cheap or in the case of HTTP/2 server push, even unnecessary. Benchmark this.
1. + 9. betrays a weird obsession with naming. The advice is not wrong, but it shifts attention away from the genuinely useful areas that benefit much more from a careful design: the media types, the hyperlinks between and other hypermedia mechanisms for traversing resources (forms, URI templates). If an inexperienced programmer follows the advice from the article, he will the idea to spend lots of time mapping out resources in the identifier space and methods and possible responses for documentation purposes. This is both a waste of time because the single entry point is enough and the rest of the resources can be reached by querying the server via OPTIONS and traversing hyperlinks etc., and dangerously brittle because of strong coupling between the server and the client.
Web Devs (such as me) have asked for e.g. DELETE as an acceptable formmethod (e.g. https://github.com/whatwg/html/issues/3577) however WHATWG always pushes back citing security concerns such as CORS.
I suspect this is not what OP had in mind since it is trivial to send a DELETE request with a simple JavaScript:
<button onclick="fetch('/api/resource/42', { method: 'DELETE' })">
Delete
</button>If CORS can be weakened in any simple way with that HTRP-DELETE method, then your database could simply disappeared via HTTP-DELETE method.
Besides, webmasters’ HTTP DELETE method is a different domain scoping issue than the web developers’ HTML/JavaScript FORM deleteThis row-entry approach.
I marvel at designers trying to flatten the scoping/nesting of abstractions without factoring apart the disparate error and protocol handling.
What is the argument for not supporting them?
The HTML forms only support GET and POST. Try it.
The recommendation is perfectly good for any API using HTTP as the substrate. The wisdom of using HTTP as a protocol substrate is questionable, but having made that decision the verbs it supplies work perfectly well.
Incidentally, the HTML living standard supports three form methods, not two: get, post, and dialog. Which rather reinforces the point that HTML != HTTP.