Hacker News Reader

Comment by iamsomewalrus | Hacker News Reader

There’s no real policy that I’m aware of internally for teams to use microservices. Amazon has a lot of tooling to make it easy to spin up services, however.

The first major project after being acquired was to make a pared down Goodreads experience available on the Kindle Paperwhite. Our first services came out of that initiative to provide a buffer between the Kindle traffic and the Goodreads Rails app.

That being said I’ll be the first to caution small teams should avoid microservices at first for fear of creating a distributed monolith.

View on HN

> That being said I’ll be the first to caution small teams should avoid microservices at first for fear of creating a distributed monolith.

What is your heuristic for determining when it is the right time to look at microservices?

That’s “the question”. I will do my best to bluff through it.

Engineering team size: you have teams large enough (~3-4) to own and iterate on a subset of related functionality for the long term.

Tooling: you have a builder tools type team that provides a tooling and observability happy path.

Traffic scale: you have functionality that operates at 2 or more order of magnitude higher that the rest of your application.

Decoupled: you have functionality that can be decoupled from your main app and, most importantly, isn’t required for your main app’s uptime. Like a search service or something.

A combination of the above.

Are internal Amazon projects cross charged for their AWS use? Or is it essentially free for internal use?

Former Amazon employee here.

They're cross-charged with lots of internal book-keeping. The rates teams "pay" internally are different from public pricing of course but expensive things externally are still expensive things internally. The cost-accounting used to be (3+ years ago) much more "just looking...no pressure to keep them down", but recently there's been huge efforts to bring internal AWS costs down, especially for EC2 usage. I heard rumors that much of the Prime Day fiasco this year could have been avoided if teams would have been permitted to spin up enough capacity.

I'm reminded of a story my ex-Amazon friend would tell where it was a week-long process to get an extra $100-something stick of RAM for his workstation when he started there, needing various sign-off and escalations for such a grave expense.

Meanwhile he was deploying some machine-learning categorization stuff he was working on to some cluster that would cost five figures of compute each time, and nobody batted an eye.

This is a pretty common thing within Amazon. Internally it's called being "Frupid" (frugal + stupid). It's one of the reasons I left.

Also you have to jump through pretty extreme hoops if your hardware estimates (usually made at least 3 months out) were under-shot and now your service is redlining. This leads to teams way over-estimating their hardware needs and thus millions being spent on idle reserved EC2 capacity. So now they police (with savagery) idle capacity, so services or workloads that are "bursty" are basically an internal-bureaucracy nightmare.

It's much nicer to use AWS outside of Amazon :)

Thanks, I really appreciate the insight.

Come join my team and I’ll be happy to spill all the beans!

Do you guys have internships on the Goodreads team?

Absolutely! We have internships in San Francisco and Seattle. Usually, they coincide with the summer, but we on occasion have Fall interns.

Reach out to me feeneyj @ amazon if you'd like I can forward you to the right people.

Reply on news.ycombinator.com