The “API Mandate” memo at Amazon
chrislaing.net
chrislaing.net
And from all the comments on this post, it sounds like that’s a pretty rational course of action.
Everyone at AWS ran around with their heads cut off back when there were 6 regions because everyone was so overworked or had to drop everything they were doing for a week because a new Xen bug was published to the email list and would go out of embargo at the end of the week. I can only imagine how insane it is now with ~20 regions to support.
And yet, there's another big tech company, fruit symbol I think, that has played the long game of continually fine tuning the interaction points to make the integration of their parts more than the whole. They get credit for attention to detail, leveraging their integration and their ability to move in ways they do because of their thorough integration across their whole hardware/software stack. They've recently been in the press because of just how much they were able to tune their integration to the problem with their first desktop computing chip. Again, we credit their success with some causality due to this strategy, which (to me) is very different than the Amazon strategy.
Is it because Amazon is a cloud company and Apple is a hardware company that they have these different approaches to product development and each enjoy success? Or at the end of the day, are these "do it this way" less responsible than we'd like to believe?
Apple's focus on design allows it to charge higher for its products and to build new high-margin products that people end up buying. It allows them to "scale" revenue by entering new markets with new products.
Amazon has a similar focus on the customer which is centered around customer support. This approach also gives them the "brand reputation" to build new services (that businesses will pay for). You may note that Apple's approach works better for customers who pay without much planning/budgeting (like consumers/households), scales better with the number of customers (again like consumers/households), and scales more poorly with # of products (making it a worse fit for SaaS).
Amazon's API mandate value is felt in how it allows them to make software development more efficient. Customers do not feel the impact directly. Instead, since data is exposed through well-defined APIs, new service (or product) development can be done with far less human communication, as mentioned in the article. However, while this makes inter-team efficiency better, it reduces intra-team efficiency by forcing developers to build things that they don't need. If services are too small, the APIs are not high-quality, or the service boundaries change too frequently, then it's possible that this approach doesn't make software engineering more efficient at Amazon.
Except that for AWS, paying for the service(s) doesn't get you any support at all. So this idea about Amazon's behavior on its marketplace doesn't really map to AWS. Customers there have to choose to receive customer support, and pay for it separately.
You can't build an OS or a chipset the same way you build AWS or a warehouse.
> it’s probably the most important single memo in the history of business
Maybe that's what you disagree with? I also find it a bit overblown. The memo worked wonders for Amazon, I'm sure it would work wonders for many other companies, but the world is more diverse than that.
I think this principle scales most to platforms, in this case web ones.
There's an interesting thing called the "Bill Gates line": https://stratechery.com/2018/the-bill-gates-line/ (third sub-heading in the article). Basically: "A platform is when the economic value of everybody that uses it, exceeds the value of the company that creates it. Then it’s a platform."
From your example, the Fruit Company is a platform, but not really. I'm not entirely convinced that others capture more value from the Fruit Company ecosystem. Yes, for the AppStore they only get 30% of the value of third party sales, but they also sell the hardware and have their own extras. The Fruit Company is a platform company primarily by virtue of creating hardware, which generally makes one a platform, but other than that, they definitely look like they don't want to be a platform with the way they control everything, break stuff, etc.
AWS is for sure a platform.
I worked at one of the first remote dev centres of Amazon working on an extremely ambitious AWS service. The mandate was imposed with a hammer in the earlier days, to an extent even the code repository was segregated. We couldn't access any internal service, no tools (such as pager-duty). So we ended up building half-ass version of everything ourselves. People at the HQ built a web-service to access customer information but within a few months it languished with no one to maintain. Half the time we would be blocked for someone to give us access to a service. Whenever we raised an issue the answer from HQ was oh yeah use this and this service, they were oblivious to our limitation. We would then play the broken record and they would then go oh, well let's see what we can do. Eventually someone higher up noticed the massive inefficiency and said fuck it and opened up all the access for us. But then everything had to be migrated from half-ass services to the mainstream ones.
During my time there I never heard of this API mandate. Now that AWS has gotten massively successful this API mandate gets paraded as if it was all a grand plan. No, it wasn't. It was an experiment that sometimes worked and sometimes didn't.
Also there's this troupe about AWS you keep hearing. The story goes that someone realised all the un-utilized server capacity and decided to rent it out. That's absolutely not how it began. That's a typical Amazon marketing speak. Amazon's retail took a very very long time to migrate to AWS. In fact I'm not sure if they are fully on AWS either.
Then Amazon realized that they could use a similar system with external customers. They built S3 and other services in a way similar to their internal infrastructure, but completely separate.
Only years later Amazon started using AWS for their internal services.
They can all be successful if you have the players executing their roles well. The structure and plan is necessary but it is not sufficient for success.
There was a recent article about another concise memo making the case to Steve Jobs for an iOS App Store, which Jobs quickly approved.
So I think the same kind of systems thinking and understanding technological ramifications of decisions at the very top, also played a large role in Apple's success.
No API, no bonus.
What followed was 18mths of people building enterprise messaging systems and "buses" of various types. Hundreds of pages of documentation on APIs was prepared and released.
I think 2 API's with maybe 40 methods actually went live.
There were no bonuses
Many people left
The people who stayed were unhappy
It never happened, everyone quietly forgot about it after 18 mnths.
The CIO stayed for another 3 or 4 years - he only left when a new CEO came in due to internal promotion. The new CEO used to run one of the P&L's/business units and hated the CIO with a passion.
So... it's not the API mandate that made the difference at Amazon.
You can't just have a free for all mentality when building APIs. The platform has to be there first.
Everyone's API should just need simple descriptive json files and then a client can be generated to consume that API easily. Everyone can declare those json files and it would work everywhere across the business.
This free for all with no standardization is bound to be a disaster.
But I would claim that this is an existence proof - if the management will and focus is there for an API enabled business (it was) this case study proves that is not sufficient to create the kind of company that Amazon became. An API program isn't a magic bullet (as everyone knows by now I guess!)
- An API program isn’t a magic bullet
- An API program is a necessary, but not sufficient condition to replicate Amazon’s success
You’re right that this case study proves that it alone is insufficient to create the kind of company that Amazon became, but does it also prove that it’s unnecessary to create the kind of company that Amazon became?
I think that the testamentary on this thread from current Amazon employees that they have ditched API requirements is evidence that it is also unneeded to sustain the kind of company that Amazon is.
in time, two-pizza teams evolved into single-threaded leader (STL) teams, a term borrowed from computer science that means to only work on one thing at a time
https://www.inc.com/jeff-haden/when-jeff-bezoss-two-pizza-te...
I am experiencing the polar opposite of this mandate. The systems in my organization are always built to require human touch-points. What's worse, our CTO mistakes these menial interactions as "teamwork" and "collaboration" when they are really just toil to compensate for the lack of platform-level thinking. I love how a CEO can put this so bluntly, upend everyones work for a couple of years and build a juggernaut because of it.
The idea runs parallel to one I have been championing throughout my career with SOAs which I call "self serve architecture". I want others in the organization to be able to pick up use my services to their benefit with zero input or help from me or my team. I tell my team to design the API as if it were GitHub's API.
Practically, that means - There are up-to-date and easy docs that cover what you need to know. - People can gain access on their own (via some existing workplace/team based credential). At most we have to add them to a list somewhere. - The system will protect itself and inform users against problematic use (quotas, throttling, and visibility into this) - You have visibility into who your users are and what they are doing such that you can assess value, learn from usage, and communicate to consumers when necessary.
"There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever"
Was Bezos deeply enough involved in Amazon's engineering to set those rules himself, or was the text of the memo influenced by a senior engineering group that he was working with?
“graduated from Princeton University in 1986. He holds a degree in electrical engineering and computer science”.
Of course he’s more known for his decision-making ability as an executive but he clearly has a solid understanding of computing fundamentals as his memo on Amazon S3 characterized it as “malloc for the Internet” [0][1].
0: https://aws.amazon.com/blogs/aws/eight-years-and-counting-of...
1: https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-...
And his first office was literally for one desk. He clearly did some of the tech work himself.
Not to be too critical, but this blog post is a (in my opinion, poor) rehash of Steve Yegge's infamous Google+ post which he accidentally posted online. It's entertaining and one of the most influential memos I've read in the last twenty years.
His followup memo was great as well.
Here's a mirror of the original essay: https://gist.github.com/chitchcock/1281611
Ha, ha! You 150-odd ex-Amazon folks here will of course realize immediately that #7 was a little joke I threw in, because Bezos most definitely does not give a shit about your day.
> #6, however, was quite real, so people went to work. Bezos assigned a couple of Chief Bulldogs to oversee the effort and ensure forward progress, headed up by Uber-Chief Bear Bulldog Rick Dalzell.
He does have a CS background.
The "even if he didn't" you quoted already implies this.
My business area uses around 200 repositories. APIs aren’t really optional at that scale.
Shopify has a component boundary interface in Ruby that can be reflected into with GraphQL, and a lot of features are built for GraphQL first anyway. First party client side apps demand it. A lot of internal services use GraphQL to talk server-server as well.
However, there's still a good chunk of monolithic logic left that hasn't been refactored yet. Refactoring efforts are mostly JIT when demanded.
These legendary Amazon Memos haven't leaked, unlike Billg's various memos [2]. That is... journalistically unfortunate, I would say. I even wonder current Amazonian actually has the access to these docs.
[1] https://www.allthingsdistributed.com/2019/08/modern-applicat...
[2] https://lettersofnote.com/2011/07/22/the-internet-tidal-wave...
> Frameworks are easier to setup initially, but they do not scale. Why is it that Microsoft Windows has 13 different dialog generations? Because each is a framework on top of a framework on top of a framework. It's amazing that they can even get that done.
> On the other side, OSS is generally built on libraries. When the 2 UNIX devs were in a basement building UNIX and were able to out-compete Multics[1], they did it because they were building libraries that could talk with each-other using pipes around the boundary. Applications that communicate based on input-output with no internal state behave just like pure functions do. Pure functions compose. When Linus built git in 10 days, he was able to do this because the core idea of git isn't actually that much work. The library is built out of composable blocks that neatly come together. Microsoft's TFS Source Control is a framework that acts on your behalf and therefore the bigger the project gets, you need n^2 people to work on it.
Amazon's API mandate is the same exact unification as UNIX's pipe effect was to Operating Systems. Amazon's internal teams are building libraries whereas all other companies started at the same time were building frameworks. Jeff was brilliant to see this at the time, and I expect that this memo will be seen as just as pivotal as the Toyota Production System, and it might already have that prestige to some. Unfortunately for other companies, you can't retrofit into it and rewrites are always a terrible idea[2].
[0]: https://news.ycombinator.com/item?id=27570917
[1]: https://www.youtube.com/watch?v=3Ea3pkTCYx4, thanks to this HN comment(https://news.ycombinator.com/item?id=27494671) for this reference.
[2]: https://www.joelonsoftware.com/2000/04/06/things-you-should-...
The Golden Rule of Platforms, "Eat Your Own Dogfood", can be rephrased as "Start with a Platform, and Then Use it for Everything." You can't just bolt it on later. Certainly not easily at any rate -- ask anyone who worked on platformizing MS Office. Or anyone who worked on platformizing Amazon. If you delay it, it'll be ten times as much work as just doing it correctly up front. You can't cheat. You can't have secret back doors for internal apps to get special priority access, not for ANY reason. You need to solve the hard problems up front.
This is gold :D
> Google engineer Steve Yegge was trying to start a robust internal discussion, not post a viral hit, when he published a 4,570-word self-styled rant about what he sees as the company’s greatest flaw to Google+. Unfortunately for Yegge, he didn’t check the settings and shared his view on Google’s failure to grasp platforms over products — including Google+ — with everyone.
> He later pulled it down on his own accord but he and Google aren’t asking that the copies already spread across the net be deleted. You can read the full post here and here, among other locations — and you should to get the real flavor about why Yegge thinks the company that does nearly everything right gets this fundamental so wrong. But a large chunk is also about his former employer Amazon, what it does wrong and how Jeff Bezos — Steve Jobs “without the fashion or design sense” — got it so right.
Source: https://gigaom.com/2011/10/12/419-the-biggest-thing-amazon-g... (Note, I think the post I'm citing might be incomplete as on my screen it stops at "Playstation Network" while the Gist continues...)
Which is a big ask, since
> The problem is that we are trying to predict what people want and deliver it for them. You can't do that.
It's like, if we hit upon something useful, it better be available as a network service with a well defined interface from the start. And I do remember looking around at how things were, and thinking, yeah, Google could definitely use some of that philosophy (without, all these years later being able to cite any specific examples). It definitely felt like it hit home at the time.
Hey do these other platforms have developer support? Developer support is sort of the accessibility for developers - Google doesn't have it. MS definitely has it. I'm thinking Amazon does too but I try to avoid them.
I wonder how this is accomplished for event driven architectures built on shared event buses.
The memo does mention pub sub, FWIW.
When there's a case for tighter coupling and less services, (and yes, there are cases for it), this memo gets brought up and microservices win the argument.
If you have a team of 5 people, launching 50 services is probably not as efficient as 10 services.
Each of my APIs is a self-contained project, with its own lifecycle.
That can, potentially, add a lot of overhead, and “concrete galosh”[0] to the project.
And I am not a fan of dogma, in general. I like flexibility, and dogma is anathema to flexibility.
[0] https://littlegreenviper.com/miscellany/concrete-galoshes/
More like we're not really big enough yet to justify multiple services, since everything still runs on a single beefy machine and we don't have enough experience running the system yet to really know where to put the service boundaries.
So how does Amazon manage this btw? I don't find AWS SDKs to be all that consistent or inconsistent. They are usually good _enough_. Is that all it takes?
By contrast, Google seems to spend a lot of time on their single repo, build the world, approach. For the most part it seems beloved, or at least people try to recreate it outside google with things like Bazel.
I feel like Google's approach is more popularized. Is that because its actually better, just advertised more, or simply consistent enough to explain?
Can anyone shed light on Amazon's approach?
I mean, the engineering architecture being described in this memo is, basically, microservices. That's certainly an extremely popular -- I would go so far as to say even vogue -- pattern for solving the problem of building software at scale.
Microservices usually overdoing it, when you have multiple microservices per team, and sometimes even per developer.
Also, microservices sometimes implemented incorrectly, where they're still communicate via shared databases, instead of encapsulating them and exposing them via service APIs only.
This Memo was born because of the real business need, while many modern microservices deployments are the result of cargo-culting GAFAMs/FANGs.
Many cargo cult web services of today can be written in Sun RPC APIs.
But yeah it is C and its IDL isn't as cool as gRPC proto files.
Their APIs are... correct. But have things like usability as a last concern. You also find some things not behaving exactly as you would expect and some things that are barely explained.
I think there's a saying there: "it's better to have two of something than none".
This can often result in inconsistency and duplication.
When that happens, a team would later be formed to unify things if needed.
Most memos from two decades ago aren't still followed.
Imagine you are a newbie who just joined a project that has successfully done what Jeff said in the memo. The success of undertaking that would have meant that your colleagues will want to continue doing this (threat of firing or not :) ) and pretty soon you'll get sucked into it and hopefully see the rewards of such design.
Soon, another team notices your team consistently getting things right and getting rewards so they have an incentive to follow (social proof). This becomes department wide next and so on. This is where momentum comes into picture. It is 2015 (let's say) and there are a dozen new departments. All of them reasonably want to get going quickly so they take up patterns that worked for other in the org. 6 more years pass by with more successes and there's no real incentive (at-least org-wide) to do something different to the one that works. My educated guess: The memo is still followed.
Yes, absolutely. Arguably even more so now that Lambda and ECS/Fargate have reduced the cost of standing up a simple service.
CORBA is quite close to direct linking, with a network in between. The developer does not see it as a service or protocol, but a library call, which is rather the point. And it's not very compatible with the next one:
> All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.
CORBA/COM never played well over the internet.
Their are now back via the gRPC fashion wave, until something else "improves" it.
I think this works much better when each team is working on what could be regarded as a complete end-to-end product e.g. a database. It works poorly when each team is working on part of a product.
That could have backfired pretty badly, haha.
> Teams must communicate with each other through these interfaces.
I read that as interfaces to the team, instead of interfaces to software the team is responsible for. Something like Mechanical Turk(https://www.mturk.com/) but for internal team communication. I'm not sure if that was what was meant but that sounds interesting.
Then again, I don't know how public availability would change the API design really...
- security
- preventing abuse
- API Anti-Corruption Layer
- sanitizing inputs and outputs
- i.e. not exposing DB IDs/PKs, or pagination cursors directly
- versioning, backward- and forward- compatibility, deprecation strategy
- usability, DX, Documentation
- reducing bandwidth use:
- caching
- eliminate over-fetching
- efficient wire formatIts all command line applications and browser interfaces (the stuff first-year developers are most familiar with building, I suppose). I was familiar with this memo so it was quite a shock. Wish they'd taken their own advice.
APIs (and opaque modules, in general) are key to the way I develop software. It works well.
(0) https://littlegreenviper.com/miscellany/evolutionary-design-...
Sounds about right
Azon turned throwaway compute instances and one-click 2 day delivery ecom into huge cash cows.
They aren't some golden example of how to properly design infrastructure, it just worked out for them. luck/timing/reinvestment had a lot to do with it.
Does it include e.g. the team writing the on-device text renderer for a Kindle? What would such a service interface look like?
This reminds me of how App Store used to ban game emulators interpreting ROMs downloaded from the internet, while at the same time allowing XML parsing.
Does it really matter if the interface you're talking to over the network is code written by another team, or data written by another team? I suppose implementation details are often hidden away in data stores, but does it have to be that way?
Yes, because it means the depedencies between systems will consistently be API dependencies, not a mix of API and datastore structure dependencies, which means that the other team will only have to reduces the constraints on change.
This pretty much freezes the database schema, the team owning it can no longer change it, because they don't know how the users use it. It also limits the possibilities to provide optimizations on the data (like caching).
What a colossal twat.
Let me break it down for you - Amazon bet the farm on this strategy and it worked out amazingly. Not following this strategy is equivalent to sabotaging the most important thing the company is doing.
While I suspect this quote is tongue in cheek, it SHOULD be a fire able offense for someone to ignore company strategy because "they know better" or are too lazy or whatever.
Dont agree with what the bossman said?: "You're fired!"
And the people love it so much, they even democratically elected the poster child of that catchphrase.
But everything else? Highly undemocratic.
Private enterprise rests on property rights. Thus Bezos, as owner, was free to write that memo, including spelling out the consequences of sabotaging the company's strategy through insubordination or incompetence.
The only way this is dictatorial is if people are coerced to work there without a legal right to quit at any time. Their employment contracts given them that right (I assume) and they spell out the consequences of quitting without giving proper notice---which one could do. Thus, they're employees, not slaves in a dictatorship.
That's not to say that Amazon or other large corporations don't have problems with mistreating workers. In fact, thinking of businesses as machine systems may encourage a mindset among management that risks dehumanizing the people who do the work. And this problem is far broader than amazon.
However, dehumanizing workers isn't inherent in private property and owners' rights to run their firms as they see fit (within the constraint of law). Look at the Guinness company - privately owned, yet a pioneer in treating workers really well, and people have tasted the quality of their work round the world. Guinness believed people have inherent dignity (as a Christian he knew they were each made in God's image), so as a business owner he knew it was good for them and for his business to treat them well.[1]
Dehumanizing employees is often a result of misaligned incentives in the legal system, of unjust laws that don't fit with reality, or more fundamentally a result of the deep levels of brokenness that exist in every human being. No one is perfect, and no human system is flawless. The distortion of private property and resulting authoritarianism in business that you ask about is a sad result of what the Bible calls sin.
[1] https://www.amazon.com/Search-God-Guinness-Biography-Changed...
That's just a bunch of random words you're saying. Reality: we're building a company so we can create wealth and support our families and achieve something. We have a plan for doing it. If you're not following the plan and just sabotaging everything, you're doing exactly that - sabotaging.
This is different than saying "hey, I don't agree, let's discuss and challenge the plat" - that's great and admirable and you should do that (and I suspect you don't actually do that in your workplace). But to silently read the strategy and then say "ah fuck'it doesn't apply to me" is a huge fuck you to your colleagues. You have chosen to accept the job and the mission, do the job.
Dictatorships are great when the leader is perfect. Problem is that they never are.
Whose families? Most private companies state very clearly that mission is to "maximize shareholder's value". That's the whole point of being privately owned. Hiring and paying employers is a necessary evil.
That's why they need legal regulation, public supervision, even whistleblower employees.
The point was that this was not up for discussion. Sure, there are less twatty ways of phrasing it, but at least you know where you stand.
You'll have every engineer complaining and stop working to fight their freedom and finally the change has to be reverted. And 5 years later, oh my god, Amazon is doing that, we need to move to that direction as well...
I imagine now that somewhere in amazon there is a "qsort" REST API that does all the sorting.
If you want a team supporting that qsort function,you want it behind a rest api