Don't Build APIs
ceklog.kindel.com
ceklog.kindel.com
When I wrote this post, I didn't imagine so many people would react by saying, essentially
"Screw the customers. Just break 'em."
One of the reasons I have my panties in wad over this topic is the Web has made it far easier for people to create APIs that get used by others. Back in the olden-days you HAD to be Microsoft or similar to get the kind of traction a kid in his parent's basement can get with a little Python and a Heroku account.
But something else has changed along the way: Composability.
Yes, I worked on COM, OLE, ActiveX and all that crap. I was thinking deep thoughts about composable software back in the stone ages (and even then, it had all already been done by others smarter than I in the 70s & 80s).
But today it is REAL. The Web technology stack has actually, finally, enabled massive distributed systems composed of loosely coupled components from multiple independent parties! The dream of my computing youth has become reality!
These things are connected by APIs.
(Which, by the way are not just function calls. A API can be a REST end-point, a file format, or a database schema, amongst other things).
Yes, you as an API provider can choose to deprecate your API anytime you want. Use kill-dates, warnmessages, etc... You can even create new, independent & parallel versions. It will help.
But you will find that someone else has built something that uses your API, that exposes its OWN API and now there's a 3rd party indirectly dependent on you. Be prepared for it.
APIs are actually easy to build. That is the problem. The solution is to realize that statement is actually false.
I consider myself to be an ok writer with a reasonable vocabulary.
I never knew it wasn't pouring.
Consider me educated. Thanks. :-)
First, in a web environment, one is probably logging all API calls and their effects on the server side. With Windows, you couldn't do that as your users were hitting local code rather than an MS server. This makes deprecation of older APIs much easier, in part because you can see which of your functions a particular API user is calling and in which order. Done right, these backend metrics will make it easy to identify the most popular use cases as well as functions that aren't used much.
Additionally, there was an interesting HN post from a few days back on self-describing APIs[1]. If this catches on, it will also make it easier to deprecate old APIs, especially if the OPTIONS request returns JSON. With a little bit of reflection up front, client code can then determine the most up to date way to achieve the effect they are trying to get with this API call. This may slow down client code (requiring one or more redirects up front) but won't crash it. Apple's support of Rosetta for PPC shows that slow emulation is a good intermediate between outright breakage (a la Facebook) and infinite grandfathering (a la early MS).
If Microsoft break their API's, then people might not upgrade their versions.
Will people stop using Twitter if they break their API's? Definitely not, so why should they bother if some third part have to pay to fix their code?
It's still a great story though.
One quick example:
In CUDA, you have to explicitly copy memory to and from the GPU. We have two basic kinds of memcpy functions--synchronous and asynchronous. Asynchronous requires some additional parameter validation because the GPU has to be able to DMA that particular piece of memory, etc. After we had been shipping this for a release or two, we noticed that our parameter checking for the asynchronous call was missing one very particular corner case and would silently fall back to synchronous copies instead of returning an error. We thought, okay, let's just fix that by returning an error because surely no one managed to hit this.
Absolute carnage. Tons of applications broke. This particular case was being used everywhere. It provided no benefit whatsoever in terms of speed; in fact, it was just a more verbose way to write a standard synchronous memcpy. People did it anyway because... they thought it must be faster because it had async in the name? I don't know.
In the end, we made the asynchronous functions silently fall back to synchronous memcpys in all cases when the stricter parameter validation failed.
It's nice that Nvidia has CUDA so they can go ahead and expose functionality in new GPU's without having to (first) deal with OpenCL standardization. However, for the long term, it would be better if we'd stick to OpenCL so at least parts of source code can be shared between CPU's and GPU's of different vendors.
As far as I know, there will be no revolutionary changes in GPGPU programming in the immediate near future (apart from badass-er GPUs).
When designing APIs, use versions and have a kill date in place. Even if you don't change the API, release the same one under a new version number. Kill access to the old version on the kill date. Keep N versions accessible at a time, to reduce the burden on app writers, but don't slack on the kill date. This will give you a timeline and procedure to avoid hacking in crazy backwards compatibility, and targets for total rewrites.
Yes, people will still complain. It's OK though if you provide a reasonable balance.
You know what happens when you get kill dates? One day all of a sudden half the web will stop working. There's a reason why people start back flipping to support out of date calls.
Customers don't care why your software just broke or whose fault it was, all they care about is it broke.
It's also an unsustainable business. How long do you think MS will be able to maintain backward full compatibility? 100 years? 200 years?
> Customers don't care why your software just broke or whose fault it was, all they care about is it broke.
Customers aren't engineers; they don't know better. We do. In any, it's going to be cheaper to abandon the stubborn customers than it will be to maintain decades worth of backward compatibility, at some point.
What's the option? Throw whole platforms out and start over every 15 years? Does that scale? It has so far only because the industry has been growing (more smartphones than PCs, more PCs than workstations, more workstations than minicomputers, etc...). But it won't grow forever.
Heck, if you want to be really hand-holdy about it, have an warnmesage attribute in your returns that mentions various issues, and start throwing deprication warnings some $time before the kill date as a reminder.
And the number of bugs and security holes coming directly out of backwards compatibility in MS products was a very big issue for a very long time. Lately they have been doing much less in terms of backwards compatibility (e.g. run in "compatibility mode"), probably as and outgrowth of this.
In my post I tried to make it clear that backwards compatibility is a challenge for successful APIs.
However, I guess if you don't want to make your APIs successful you can do all kinds of things that make them harder to use, like putting in draconian kill-dates, warnmessages, etc... Make sure your documentation is extra-long and wordy too while you're at it.
Because, sophacles, is right: if you don't have a successful API you don't have to worry about pissing customers off.
Having a reasonable end of life on support never hurt apple. There are lots of things that I can no longer run on the latest version of OSX. This happens every time the OS upgrades. Except of course they don't use versioned APIs, so it is much less planned for and not really as easy to deal with for devs.
I'm not saying make a shitty API, I'm saying, if you have the people who depend on your product know that changes happen, they can prepare for it. If you release a new version of the API every 6 months, and only support them for 18 months, you can make the api migrate slowly enough that it isn't a serious change for each version.
In native code: anything compiled to old APIs can link to old versions of your library, and just work the same way. In web apis: the world changes faster than the 18 month thing anyway, so whatever.
If people want to depend on your bugs, you told them not to, they can deal with the changes.
Just because the iPhone is doing great, doesn't mean businesses and other slow movers avoid Apple because of this. My point is, we don't know if this hurts Apple or not.
This is what i hate about osx - they freaking change the OS drastically, and my apps break. Sure, there might be a new version, but when i compare mac OSX to windows, windows kicks ass because you know what? i can _still_ run crap that i used to run in win95!
But your proposal is interesting: if things are improving all the time, API-users need to be upgrading all the time and adapting to the new features anyway - so any incompatibility necessary for those features is no problem.
It depends on the phase of market (new, growth, mature, stagnating) for that specific API. At the moment, most web-APIs usages are "new".
Obviously the needs for a web API and the needs for an API with a specific binary ABI on a local OS are quite different but I think for either environment, most API developers can comfortably fit somewhere between those two extremes where they don't continue to support inherently unsupported API usage, but they don't break something randomly every week.
As a developer, it means we don't have to keep digging out the old projects and porting them to the new API.
As a user, it means we are left with systems that have many years of cruft and with features being held back from systems because they couldn't make it backward-compatabile.
I think most users are willing to pay any higher price that will result from developers being required to work harder to port their software.
What is the "myth" of backward compatible?
I wholeheartedly agree with your second paragraph regarding versions and kill dates - but every API is different. In our case, when our API changes/versions/deprecates - WE are directly held responsible by customers of consumers of the API - since API consumers trumpet the fact that they have integrated with our API.
On the surface, versioning and kill dates are great end-all solutions, but the reality is API consumers have a ton of leverage when your API is successful at generating revenue for you. Especially when those consumers are sending lots of revenue your way from their own customers, which think you are the source.
Don't do that. That way lays insanity. Be very, very clear up front that you will break backwards compatibility for those folks. Don't sweep it under the table, be very vocal about having done it. There will be short term pain as important customers (eg Adobe) learn the hard way that you really mean it. And long term relief as you don't have that legacy headache growing so quickly.
One estimate is that a 25% increase in requirements results in a 100% increase in software complexity. (See Facts and Fallacies of Software Engineering for a source.) That means that the complexity grows as the number of requirements cubed. Therefore the key to a good API is that it be short and simple. When you start to add implicit requirements on top of explicit ones, it quickly becomes long and complex, and the necessary software to support it and make future modifications becomes much worse.
This does not mean that designing APIs is not hard. But don't let your API become what is published and quirks that are not. Just don't.
I think this is very naive/utopian.
Yes, be super focused in your design.
Yes, only expose APIs you have clear use cases for.
Yes, keep the surface area as small as possible.
Yes, have very focused requirements.
Yes, document the hell out of things.
Yes, implement strong parameter validation and other things to try to reduce the chance people do bad things.
Do all these things and more (these are all part of what makes exposing APIs hard work, that many people don't do).
But do not, for one second, believe that someone still won't do it wrong or abuse your perfectly designed API eventually... especially if it is successful.
Microsoft failed at keeping the surface area as small as possible. When their surface area expanded to a hack to deal with Adobe's hack of replacing code behind Microsoft's back, they went into very dangerous territory.
Even if noone will ever use all three of the broken apps.
In this sense, maintaining backwards compatibility is simply a matter of reputation management.
Not a chance. People will blame Microsoft, because WhateverIUse worked perfectly before. The only parameter which changed is that they updated the OS, therefore the OS is to blame.
The flip side if Microsoft had bitten that painful bullet up front, then I believe that going forward companies like Adobe would have learned to not try to hack the system like they had. That would have given everyone, including Microsoft, a better reputation.
But of course Microsoft did not do that. And now Microsoft has such a history of nasty bugs that, even if Microsoft is in the right, nobody is going to believe it. (Which of course locks Microsoft into its current strategy of bending over backwards to accommodate everyone else's bugs, causing more bugs...)
It's not just an issue of parameter validation or well-defined calls; it's the interactions between calls where this is hard. It's issues of ordering, timing, when you can secretly skip synchronization because you know the API will synchronize just enough for you to get away without something more heavyweight, things like that. And then when you have paying customers? Yeah, you can't break their code, even if it's bad, because if you do you won't have customers for very long.
Behind almost every successful API is a big ugly mess of backwards compatibility to keep customers happy.
FreshBooks has a very significant amount of their usage/profit from their API. These are just their endorsed/vetted add-ons, let alone all the ones out there in the wild: http://community.freshbooks.com/addons/?header_addons=1 and they clearly built an app AND an API.
The API for FreshBooks was a major portion of their (very successful) strategy, so I can't see why people can't do both, provided they do it intelligently.
But, this quote in the post is important: "When exposing APIs be absolutely certain the value you get from doing so is worth it."
In our case, our APIs are a significant revenue driver and are worth it. Don't let OP discourage you from exposing your platform's data via APIs. Instead, let this post warn you what to look out for when exposing APIs.
Are most of your API users ahead of you, or behind you?
If you have an immature API with a smallish number of users, and you think that incompatible changes in the API will improve adoption with the vast bulk of your (not-yet-on-board) potential market, then go ahead, break the API.
If the only reason people use your API is because of its legacy -- if you think that by changing your API people will wake up from their inertial slumber and investiate your superior competitors, then don't break your API under any circumstances.
Obviously there are grey points in between, and a series of small breaking changes will be worse than occasional large breaking changes. But by and large, the success of your platform will depend on its utility, not legacy compatibility. This is why stripe and wepay will eventually conquer paypal, why python will continue to be a vibrant community, why java will eventually fade to cobol-like obscurity, and why Microsoft Windows is fundamentally doomed. It's hard to look backwards and forwards at the same time.
http://www.joelonsoftware.com/articles/APIWar.html
Essentially, Microsoft shoots itself in the foot by trying to stay extremely backwards compatible, even to the detriment of making their products better.
Facebook and Paypal are or have been guilty of this in the past though Facebook is a lot better than it was.
At least you wouldn't be stuck debugging assembly code.
This was great for me because my code never broke, which was important because It was running the back-end of an e-commerce site. It gave me more than enough time to upgrade when I wanted bug fixes/features.