Everything I know about good API design
seangoedecke.com
seangoedecke.com
It illustrates that the reminder isn't "never change an API in a way that breaks someone", it's the more nuanced "declare what's stable, and never break those".
So Linux is opinionated in both directions - towards user space and toward hardware - but in the opposite way
It is a bit ironic and a little funny that Windows solved this problem a couple decades ago with redistributables.
musl libc has a more permissive licence, but I hear it performs worse than GNU libc. One can hope for LLVM libc[1] so the entire toolchain would become Clang/LLVM, from the compiler driver to the C/C++ standard libraries. And then it'd be nice to whole-program-optimise from user code all the way to the libc implementation, rip through dead code, and collapse binary sizes.
But I don't think I've ever seen anybody actually do this.
> one who thinks that distributing an LGPL library without your own app's corresponding source code is forbidden is someone who flat out doesn't understand the LGPL
Most people use musl, though some others use uclibc.
Musl is actually great, even if it comes with some performance drawbacks in a few cases.
If your program is built to require myfavoritelibrary version 1.9, and you try to run it against myfavoritelibrary 1.0, no shit it doesn't work. Glibc is no different than any other in this regard.
If your program is built to require myfavoritelibrary version 1.0, and you try to run it on myfavoritelibrary 1.9 ... glibc's binary compatibility story has been very good since the release of 2.2 or so, way back in 2000. (I know from documentation that there were a lot of 2.0 -> 2.1 breakages, some of which might've actually been fixed in 2.1.x point releases, so I'm saying 2.2 to be safe)
It's not quite as perfect as Linux's "we do not break userland" but it's pretty darn close; I would have to hunt down changelogs to find something that actually broke without explicitly relying on "do not rely on this" APIs. Source compatibility is a different story, since deprecated APIs can be removed from the public headers but still present in the binary.
... actually, even Linux has unapologetically broken its promise pretty badly in the past at various times. The 2.4 to 2.6 transition in particular was nasty. I'm also aware of at least one common syscall that broke in a very nasty way in some early versions; you can't just use ENOSYS to detect it but have to set up extra registers in a particular way to induce failure for incompatible versions (but only on some architectures; good luck with your testing!)
---
There's nothing stopping you from installing and using the latest glibc and libgcc at runtime, though you'll have to work around your distro's happy path. Just be careful if you're building against them since you probably don't want to add extra dependencies for everything you build.
By contrast, I have statically-linked binaries from ~2006 that simply do not work anymore, because something in the filesystem has changed and their version of libc can't be fixed the way the dynamically-linked version has.
https://abi-laboratory.pro/?view=timeline&l=glibc
Granted it hasn't been updated since 2023, you can still see the trend with removed symbols in each version.
* 9 malloc debugging variables were removed, though their symbols actually remain for backwards compatibility they just don't do anything. * vtimes was removed, but the symbol remains for backwards compatibility
Those were the only changelog entries listing removals. None of them cause linking issues. The 9 that did break backwards compatibility are a set of debug tools that don't work for alternate memory allocators and the functionality can be brought back with libc_malloc_debug.so.
Maybe the changelog's incomplete, but this actually seem pretty tame to me.
It has been better than most but they recently broke loading libraries that declared they need an executable stack (even if the library never used it) and there doesn't seem to be a plan to actually fix the backwards compatibility issue.
With Linux, which is a C codebase by and large, they load and pass pointers to structures to kernel procedures which can do as they please -- as long as the documentation on said structures (which usually says which fields and how are retained with which values and so on) remains unchanged. That's their "object oriented programming" (yeah, I know Linus would likely have hated the comparison).
You cannot predict the future and chances are there will be some breaking change forced upon you by someone or something out of your control.
/v1/downloadFile
/v2/downloadFile
Is much easier to check for a v3 then
/api/downloadFile
/api/downloadFileOver2gb
/api/downloadSignedFile
Etc. Etc.
It's typically to declare bankruptcy on the entirety of /v1 and force eventual migration of everyone onto /v2 (if that's even possible).
For example dup(), dup2(), dup3() and pipe(), pipe2() etc
LWN has an article: https://lwn.net/Articles/585415/
It talks about avoiding this by designing future APIs using a flags bitmask to allow API to be extended in future.
^/api(/(?!v[0-9]).)?$ is v1
^/api/v2(/.)?$ is v2
It's really not an issue in any case, it just itches your brain, because is not as neat as you would like it to be.
What actually happens as the API grows-
First, the team extends the existing endpoints as much as possible, adding new fields/options without breaking compatibility.
Then, once they need to have backwards-incompatible operations, it's more likely that they will also want to revisit the endpoint naming too, so they'll just create new endpoints with new names. (instead of naming anything "v2").
Then, if the entire API needs to be reworked, it's more likely that the team will just decide to deprecate the entire service/API, and then launch a new and better service with a different name to replace it.
So in the end, it's really rare that any endpoints ever have "/v2" in the name. I've been in the industry 25 years and only once have I seen a service that had a "/v2" to go with its "/v1".
This is an interesting empirical question - take the 100 most used HTTP APIs and see what they do for backward-incompatible changes and see what versions are available. Maybe an LLM could figure this out.
I've been just using the Dropbox API and it is, sure enough, on "v2". (although they save you a character in the URL by prefixing "/2/").
Interesting to see some of the choices in v1->v2,
https://www.dropbox.com/developers/reference/migration-guide
They use a spec language they developed called stone (https://github.com/dropbox/stone).
So you would add “v1”, to be able to easily bump to v2 later if needed, and do your best to avoid bumping to v2 if at all possible.
You don’t really need to do that for REST APIs. If clients request application/vnd.foobar then you can always add application/vnd.foo.bar;version=2 later without planning this in advance.
But thanks to versions, in my job we renamed old APIs like /v1/oauthApple and /v1/oauthGoogle to /v2/login/oauth/apple and /v2/login/oauth/google, looks so much better.
That doesn’t make sense. The whole point of creating a new version is to change the schema. And what do you mean “rename it”?
> But thanks to versions, in my job we renamed old APIs like /v1/oauthApple and /v1/oauthGoogle to /v2/login/oauth/apple and /v2/login/oauth/google, looks so much better.
Wait, by schema do you mean URL structure?
You’re looking at this backwards. The benefit of using headers is that you can keep the same URL. In a REST API, the URL is the primary key. If Client A holds a copy of /v1/foo/1 and Client B holds a copy of /v2/foo/1 then as far as HTTP and REST are concerned, those are two different resources and the clients cannot interoperate. If Client A holds a copy of /foo/1 in application/vnd.foo;version=1 format and Client B holds a copy of /foo/1 in application/vnd.foo;version=2 format, then those clients have the same resource and can interoperate.
But if you want to change your URL structure, you can do that too. There’s nothing stopping you from moving /oauthApple to /oauth/apple, you don’t even need a new version to do that – just change the link.
Cursor based pagination (using the ID of the last object on the previous page) will give you a new list of items that haven't been viewed. This is helpful for infinite scrolling.
The downside to cursor based pagination is that it's hard to build a jump to page N button.
You can do some other cool stuff if they're opaque - encode additional state within the cursor itself: search parameters, warm cache / routing topology, etc.
I'm not sure how would storing a key in Redis achieve idempotency in all failure cases. What's the algorithm? Imagine a server handling the request is doing a conditional write (like SET key 1 NX), and sees that the key is already stored. What then, skip creating a comment? Can't assume that the comment had been created before, since the process could have been killed in-between storing the key in Redis and actually creating the comment in the database.
An attempt to store idempotency key needs to be atomically committed (and rolled back in case it's unsuccessful) together with the operation payload, i.e. it always has to be a resource-specific id. For all intents and purposes, the idempotency key is the ID of the operation (request) being executed, be it "comment creation" or "comment update".
Putting version numbers in a URL is a bit of a kludge. v1 is the most common version, by far, you will ever see in a url. v2 is rare. v3 is more common strangely. I don't think I've seen a v4 or v5 or higher in the wild very often. That's just not a thing.
My theory is that v1 is the quick and dirty version that developers would like to forget exists. v2 is the "now we know what we're doing!" version and that's usually quickly followed by v3 because if you can change your mind once you can do it twice. After which people just tell developers to quit messing with the API already and keep things stable. v4 and v5 never happen.
Another observation is that semantic versioning for API urls here seems rare. Reason: it's inconvenient for clients to have to update all their URLs every time some developer changes their mind. Most clients will hard code the version. Because it never changes. And because it is hard coded, changing the version becomes inconvenient.
My attitude towards URL based versioning is that you could do it but it's not a tool that you get to use much. Therefore you can safely skip it and it won't be a problem. And in the worst case where you do need it, you can easily add a v2 URL space anyway. But you probably never will as you are unlikely to deprecated the entirety of your API.
There are other ways to deal with deprecating APIs. You can just add new paths or path prefixes in your API as needed. You can use a different domain. Or you can just remove them after some grace period. It depends. Versioning is more aspirational than actually a thing with APIs.
We do version our API but via client headers. Our API client sends a version header. And we check it server side and reject older versions with a version conflict response (409). This enables us to force users of our app to update to something we still support. The version number of our client library increments regularly. Anything falling behind too far we reject. This doesn't work for all use cases. But for a web app this is completely fine.
That way you have versioning and also simple urls.
Azure I'm looking at you. Many of their services do this, but Blob storage is something else: I've literally gotten information-free responses there. (I.e., 0 B of actual data. I wish I could say 0 B were used to transfer it.)
When you're designing, think about how big a record/object/item is, and return a reasonable number of them in a page. For programmatic consumers who want to walk the dataset, a 640 KiB response is really not that big, and I've seen so many times responses orders of magnitude less, because someone thought "100 items is a good page size, right?" and 100 items was like 4 KiB of data.
> If you have thirty API endpoints, every new version you add introduces thirty new endpoints to maintain. You will rapidly end up with hundreds of APIs that all need testing, debugging, and customer support.
You version the one thing that's changing.
As much as I hate the /v2/... form of versioning, nobody reversions all the /v1/... APIs just because one API needed a /v2. /v2 is ghost town, save for the /v2 APIs.
Of course the should be some sort of maximum size, but I have seen APIs that return 1200 lines of text and require me to page them at 100 per request with no option to turn it off.
Not implementing pagination at the outset can be problematic, however. If you later want to paginate data (e.g. if the size of your data grows) then it’s going to be a breaking change to implement that later. Big page sizes but with pagination can be a reasonable balance.
If at all possible, take your time and dog-food your API before opening it up to others. Once it's opened, you're stuck and need to respect the "never break userspace" contract.
There’s a lot of things you can do with internal users to prevent causing a burden though - often the most helpful one is just collaborating on the spec and making the working copy available to stakeholders. Even if it’s a living document, letting them have a frame of reference can be very helpful (as long as your office politics prevent them from causing issues for you over parts in progress they do not like.)
It’s not really an option in the same way with end users or customers, as they aren’t part of your organisation, by definition.
Things that's missing from this list but that were important for me at some points:
1. Deadlines. Your API should allow to specify the deadline after which the request is no longer going to matter. The API implementation can use this deadline to cancel any pending operations.
2. Closely related: backpressure and dependent services. Your API should be designed to not overload its own dependent services with useless retries. Some retries might be useful, but in general the API should quickly propagate the error status back to the callers.
3. Static stability. The system behind the API should be designed to fail static, so that it retains some functionality even if the mutating operations fail.
Not necessarily. Many implementations return HTTP 204 for any DELETE that succeeds in the sense that the element is gone regardless if it had been there before. To me this always made much more sense than 404.
The client already has the capability to handle 404 for DELETEs.
But "API" means "Application Programming Interface". It was originally for application programs, which were... programs with user interfaces! It comes from the 1940's originally, and wasn't referred to for much else until 1990. APIs have existed for over 80 years. Books and papers have been published on the subject that are older than many of the people reading this text right now.
What might've those older APIs been like? What were they working with? What was their purpose? How did those programmers solve their problems? How might that be relevant to you?
I don't see it. API is too vague to mean one type of API, whether it's one from before the web, or a web API. As soon as there was more than one type of API, the term API became incomplete without a qualifier. Nothing was hijacked, and your sentence includes an incomplete term.
Really?! That’s amazingly early. There were barely even subroutine libraries at that time. I’d love to see an example of "Application Programming Interface" from that time.
(I don’t remember seeing the term until Microsoft started using it when talking about Windows in the 1990s; before then it was things like library functions or supervisor calls - but I didn’t have much experience at that point so I was probably missing some of the more collar-and-tie programmer lingo.)
One issue I have with weird resources are those that feel like unnecessary abstraction. It makes it hard for the human to read and understand intuitively, especially someone new to these set of APIs. Also, it makes it so much harder to troubleshoot during an incident.
The quality of the API is inversely correlated to how difficult it is to obtain API documentation. If you are only going to get the API documentation after signing a contract, just assume it’s dismally bad.
This is not just true for authentication. If you work in a business setting, your APIs will be used by the most random set of users. They be able to google for how to call your api in python, but not be able to do things like converting UTC to their local time zone.
Sigh... I wish this were not true. It's a shame that no alternatives have emerged so far.
The separation of a refresh cycle is an optimization done for scale. You don't need to do it if you don't need the scale. (And you need a really huge scale to hit that need.)
If a client is accessing an API on behalf of itself (which is a more natural fit for an API Key replacement) then we can use client_credentials with either client secret authentication or JWT bearer authentication instead.
There doesn't need to be any OIDC or third party involved to get all the benefits of them. The keys can't be used by multiple simultaneous clients, they naturally expire and rotate over time, and you can easily audit their use (primarily due to the last two principles).
I never understood why.
1. Generate your initial refresh token for the user just like you would a random API key. You really don't need to use a JWT, but you could.
2. The client sends the refresh token to an authentication endpoint. This endpoint validates the token, expires the refresh token and any prior bearer tokens issued to it. The client gets back a new refresh token and a bearer token with an expiration window (lets call it five minutes).
3. The client uses the bearer token for all requests to your API until it expires
4. If the client wants to continue using the API, go back to step 2.
The benefits of that minimal version:
Client restriction and user behavior steering. With the bearer tokens expiring quickly, and refresh tokens being one-time use it is infeasible to share a single credential between multiple clients. With easy provisioning, this will get users to generate one credential per client.
Breach containment and blast radius reduction. If your bearer tokens leak (logs being a surprisingly high source for these), they automatically expire when left in backups or deep in the objects of your git repo. If a bearer token is compromised, it's only valid for your expiration window. If a refresh token is compromised and used, the legitimate client will be knocked offline increasing the likelihood of detection. This property also allows you to know if a leaked refresh token was used at all before it was revoked.
Audit and monitoring opportunities. Every refresh creates a logging checkpoint where you can track usage patterns, detect anomalies, and enforce policy changes. This gives you natural rate limiting and abuse detection points.
Most security frameworks (SOC 2, ISO 27001, etc.) prefer time-limited credentials as a basic security control.
Add an expiration time to refresh tokens to naturally clean up access from broken or no longer used clients. Example: Daily backup script. Refresh token's expiration window is 90 days. The backups would have to not run for 90 days before the token was an issue. If it was still needed the effort is low, just provision a new API key. After 90 days of failure you either already needed to perform maintenance on your backup system or you moved to something else without revoking the access keys.
And what time frame is “long-lived”? IME access tokens almost always have a lifetime of one week and refresh tokens anywhere from 6 months to a year.
OAuth flows are not at all common for server-to-server communications.
In my perfect world, I would replace API keys with certificates and use mutual TLS for authentication.
I hate mTLS APIs because they often mean I need to change how my services are bundled and deployed. But to your point, if everything were mTLS I wouldn’t care.
Both, really. mTLS deployment is the sticking point, but it's slowly getting better. AWS load balancers now support it, they terminate the TLS connection, validate the certificate, and stick it into an HTTP header. Google Cloud Platform and CloudFlare also support it.
If you have sensitive resources they’ll be blocked behind some authz anyway. An exception I’ve seen is access to a sandbox env, those are easily generated at the press of a button.
Some way to break out of the "shared secret" model is needed. Mutual TLS is one way that is at least getting some traction.
> ...You’re building it for a very wide cross-section of people, many of whom are not comfortable writing or reading code. If your API requires users to do anything difficult - like performing an OAuth handshake - many of those users will struggle.
Sounds like they're talking about onboarding specifically. I actually really like this idea, because I've certainly had my fair share of difficulty just trying to get the dang thing to work.
Security wise perhaps not the best, but mitigations like staging only or rate limiting seem sufficient to me.
For end points it’s a bit different. You don’t know what are they or user facing or programmer facing.
I wonder if someone has a good take on this. I’m curious to learn.
Why does the type of I/O boundary matter?
Maybe I just don't understand what the word interface means other than the GUI version of it. What's an interface in the analogue world? [2]
By the way, one person downvoted me. To that person: it's fine that you downvoted me, but also let's try to keep an open and inclusive culture?
I know it's a beginner question. I'm not a beginner, I use APIs all the time and have designed them as well. Just how I used servers without knowing for 5 years the semantic meaning behind it [1]. Understanding things deeply in that way is not my forte.
[1] Though most people still don't know that clients/servers are roles of computer programs. Many programmers conflate servers with actual hardware in the sense of "a computer can be a client of a server". Well, no a piece of code can be a client to another piece of code and a server can be a piece of code to another piece of code. They're roles, not distinct hardware.
[2] Claude mentions:
Analog World Interfaces
Door handle - The interface to a door mechanism. Whether it's a simple latch or complex electronic lock, you just turn/push the handle. An interface is basically the part you touch/use without needing to understand what's behind it.
First, to contrast the "application programmer interface" (i.e. what code has to be written to use it properly in an environment with a compiler) from the "application binary interface" (i.e. what actual bytes have to be provided in order to work with an already compiled version — important for linking, inter-process communication etc.).
Second, to be able to talk about what you actually do with the headers (i.e. what the rules are for using the code) separately from the headers themselves (i.e. just the code itself), and to abstract over other programming languages that don't work the same way.
> They're roles, not distinct hardware.
So, you already understand the value of these kinds of distinctions. A shame we haven't historically made them more consistently. (As an exercise, try to list everything that "static" can mean in every programming language you know where that's a keyword. For bonus points, contrast and compare to how the word is used in the discussion of programming language design.)
> Maybe I just don't understand what the word interface means other than the GUI version of it. What's an interface in the analogue world? ... Claude mentions:
In the world that I was recalling, when people were unfamiliar with a word, they used a resource called a "dictionary" to look them up. This provided a pre-written answer directly, rather than relying on sophisticated computer models to come up with something new every time. Admittedly, this did trend towards online use over time, in particular since this made it easier to update these resources to reflect current use patterns. But those online resources are still available, e.g. https://www.merriam-webster.com/dictionary/interface .
Even with AI expanding so far as to creep into search engines, you can still reliably obtain such definitions with search queries consisting of "define" + the word.
Some applications live in a single process, while others span processes and machines. There are clear differences, but also enough in common to speak of “APIs” for both
Just as I am irritated that people seem to have forgotten that "applications" can potentially run completely locally, and that programs can be designed around the assumption that the "front end" (UI) and "back end" (business logic) will communicate directly (or at least exist within the same process, even if there are good reasons to set up message queues etc.).
But, you know, that's bad for business. Because that entails that the consumer might actually get to own something, even if it's just an ephemeral pattern of bits on local storage.
The format and protocol of communication was never fixed.
In addition to the rest api’s of today, soap, wsdl, web sockets could all can deliver some form of API.
Shudder...
In my circles this is usually (perhaps incorrectly) called REST API.
I'd like to introduce more fields or flags to control the behavior as params, not asking user to change the whole base url for single new API.
When an API commits to /v1 it doesn't mean it will deprecate /v1 when /v2 or /v3 come out, it just means we're committing to supporting older URI strategies and responses.
/v2 and /v3 give you that flexibility to improve without affecting existing customers.
I worked in an org where idempotency meant: if it threw an exception this time, it needs to throw the same exception everytime.
Wait. Surely "ORDER BY id OFFSET 20 LIMIT 10" works about the same as "WHERE id > cursor ORDER BY id LIMIT 10", if "id" is indexed?
https://use-the-index-luke.com/sql/partial-results/fetch-nex...
Perhaps some of the page-level Copy-on-Write databases (LMDB?) might do this, since they have to rewrite ancestor pages anyway.
GraphQL isn’t just another protocol. It’s a paradigm shift in how we think about designing and consuming APIs. The author downplays its role, but in practice, GraphQL enables cleaner contracts between frontend and backend, encourages typed schemas, and dramatically reduces over-fetching and under-fetching. That’s not a minor point .. that’s central to what most developers care about when consuming APIs.
Regarding caching: yes, REST has traditional browser and CDN-based caching, but GraphQL is absolutely cacheable too. Tools like Apollo Client and Relay have built-in normalized caches that are far more granular and powerful than most REST setups. At the infrastructure level, persisted queries and CDN layer solutions (like GraphCDN or Stellate) further optimize caching behavior. So the claim that “you can’t cache GraphQL” is outdated at best.
I'm not sure what you mean by the react analogy, react seems to be far more popular than graphql in their respective areas.
There are performance footguns (like recursion, for which you'd have to consult your GraphQL server library for mitigations). There is often a built-in 'introspection' endpoint, which many consider a security faux-pas (to which I disagree — I think it's pretty noble, like having built-in OpenAPI docs) BUT you can easily craft a recursive query using just this endpoint that will take (some) GraphQL servers down. There are plenty of posts written on the matter and I'm sure there are mitigations, but my first exposure to GraphQL was on someone else's project (a respectable engineer who I consider very skilled) and within the first day I had noticed this intriguing structural hazard and taken the server down...
It therefore seems to be a tool that is possibly difficult to 'hold correctly'; at least people should be cautious about going in without doing their research about these things. Probably a maturity issue, but one that people ought to be aware of?
Other gripes:
It's displeasing how the HTTP status code does not correlate with the actual success response of the API, which makes typical request logging less useful.
I guess it also makes it harder to provide optimised hot paths on the server (because your client team might shift their queries around, or whatever).
In previous experience I also find that having easily-recognisable names for API endpoints (like `GET /repositories`) makes talking about them, and recognising them, easier than the more-opaque-feeling GraphQL approach.
> Be descriptive in your error responses
This is a useful standardised format:
RFC 7807: Problem Details for HTTP APIs
Versioning, etc. matter (or don’t) for binary UDP APIs (aka protocols) just as much as for any web API.
Bravo!
If the consumer wants to consume a specific version of the API, the means to do so can be implemented with an alternative domain name, or -- even better (who wants to maintain alternative domain names) -- with a request _header_, e.g. `X-API-Version: v1` (or another one, perhaps a standardised one).
In any case, the `/v1/` thing is something of cargo cult programming -- I remember someone proposed it a good while ago, and it's been adopted since without much afterthought, it seems. It doesn't make sense to debate pros and cons of REST/HATEOAS if your resource identifier scheme is poorly designed, IMO.
This is an extremely unpopular opinion, but I would go even further. I think you should let people use your API with just an username and a password.
It should by no means be the only way people can use your API. Put very low users-per-IP rate limits on that approach if you want to, to force lazy but professional software developers to go the oAuth route before their app gets to production. For one-off scripts though, APIs that let you do this are a breath of fresh air.
If your API is based on API keys, you will be tempted to do things that really annoy new users of that API. People don't want to tell you what their app name is, they don't know that yet. They're certainly not picking a purpose they need this API for from a list of five, not if it doesn't include "completing a classroom assignment I don't really care about and want to finish as quickly as possible." They for sure don't yet know what scopes they might possibly need, even if to you, their names are descriptive and obvious. If you allow user-password authentication, you take away the ability to shoot yourself in the foot in this way.