Comment by jedberg | Hacker News Reader

Comment by jedberg | Hacker News Reader

Parent Full thread

AWS has a very data driven culture, almost to a fault. They are extremely customer-centric, but almost too much so.

For any item on this list, if someone at AWS with the necessary authority thinks it's a good idea, they can't just find an engineer and have it implemented.

First, a PM has to talk to a bunch of customers of varying sizes to see if the feature would be useful to them and if so what their use cases are. Then they need to scope the project, write the press release, have it run through a committee of Principle engineers, and come up with a spec. Then they need to take that spec back to the customers for feedback.

After all that, it will finally get onto a list to be added to the roadmap in the future. Then it will get onto a roadmap hopefully. And if it's not "next quarter", there is a really good chance it will fall back off that roadmap.

Then it will finally get implemented and launched in private beta for key customers who expressed interest. Then a wider beta. And only after all that will it get released for everyone else.

I understand the need for some of that process, since when it launches it will be instantly available to millions of customers and will need to scale. But there has to be a middle ground here. Like: customer asks for feature, engineer builds it and makes it available only to that customer with the caveat that it could break or change at any time. Then iterate from there, maybe slowly adding in new customers while at the same time talking to customers and integrating their feedback. And maybe that does happen, but I've been part of some pretty big customers, and I can't think of a single time we got a feature right away that went through major changes. The best we ever got was "special important customer" just before a wider beta. But by then the API was basically set.

AWS has established itself as being highly backwards compatible - and the cost of that seems to be the long-winded process you described. If they were more liberal with what features they implement then they would have to either be more liberal with what features they deprecate (removing a core value proposition of AWS: backwards compatibility) or they would have to suck it up and deal with the bloat which would naturally lead to higher cost and overall slower delivery of new services.

For a counter example see GCP. One of the biggest barriers to entry is trust: no one wants to use it because they don't trust Google to not arbitrarily yeet features out the window with six months notice. Whether or not that's actually what GCP does, it's how people perceive Google.

I'll take AWS's approach for things that I work on professionally and have to support for many years.

GCP seems to be generally better designed though. Despite how conservative towards adding new features, amazon is being described here, I am often unimpressed at how poorly conceived many things are within AWS

I totally trust Google to arbitrarily yeet features out the window with six months notice...

The process I described would not in any way harm backwards compatibility though. They can still run the same process, but in parallel, so that iteration can happen faster amongst people who care the most.

Will that not lead to bloat and/or more deprecation of services in the future?

I suppose you can tell people "hey this service shouldn't be relied on! We might get rid of it one day!". But that erodes trust - people just don't fully internalise what that means, they expect the same level of long term support from all AWS services. Soon the common sentiment will turn into "AWS changes its services all the time" when the truth is "AWS changes the services marked as experimental all the time".

If you only give access to a few customers I don't think that would be a problem. Especially if you purposely make breaking changes to the API during the alpha period to make it infeasible to use in production.

The key is to not give wide access. Google's problem was that anyone could turn on "labs" and use them, and then get really upset when they went away. A few customers getting really early access to a feature under NDA would not be nearly the same.

It really depends on how it is rolled out and managed. I have lived, first hand, issues where you end up with two trains that end up being impossible to merge without breaking users or one or the other.

> engineer builds it and makes it available only to that customer with the caveat that it could break or change at any time.

This is likely why - you may get people to sign off on “break or change at any time” but the reality is once a customer is using it you’re going to have to support it indefinitely.

Implementing even relatively small features at Amazon takes quite a bit of effort, usually across multiple teams. The codebases are enormous, the number of services involved is inevitably at least a handful, athe side effects of a change can be hard to reason about, etc. The idea that a single engineer is going to implement a feature for one customer in some really short timeframe is just not in ballpark of reality.

And even if that were a fathomable process, one must consider the opportunity cost of assigning engineers to one-off feature work rather than on vetted roadmap features. This sort of thing is frequently done at small companies trying to save an important contract or similar, sometimes to great detriment. Amazon isn't a place where a single customer (apart from maybe US government) has such pull.

It sounds like a place where "hero" PM's who know how to navigate the system and attract buy-in would do well.

Isn't part of a PM's job generally to navigate the system and get buy in?

Depending on how navigable the system is that can be an afterthought, a major component, or their entire job, leaving the other responsibilities to fall by the wayside.

> They are extremely customer-centric

Is someone holding a gun to your head and making you say this?

What you described is exactly my experience interacting with AWS as a (sizable) customer, yes, but it's not remotely what I would consider customer centric. Bureaucracy-centric, perhaps, or bottom-line centric, but certainly not customer centric. Even getting AWS to do something for its own benefit is like pulling teeth. For customer benefit? Fugheddaboudit, unless you are huge and are willing to relentlessly hound them for years and throw gobs of cash at them and deal with the betas that somehow manage to be even more buggy, feature-anemic, and gotcha-landmined than their public-facing services.

Reply on news.ycombinator.com