The Amazon Builders' Library | Hacker News Reader

The Amazon Builders' Library

aws.amazon.com

469 pointsmooreds48 comments

Wow. I haven’t gone through most of the content but shuffle sharding not only is a great idea but their write up is quite clear: https://aws.amazon.com/builders-library/workload-isolation-u...

Impressive stuff!

Yes, that's a great one. But now I have a brain teaser stuck in my head: Imagine you have a multi-tenant system like they describe, but for each customer A, B, C they have users A1, A2, A3, B1 etc.

You need ordered delivery of messages, but only at user level (if all messages needed to be ordered you could not do this sharding type setup). Is there a way to apply shuffle sharding while maintaining user-level ordering guarantees?

Yes! we do recursive, nested, shuffle sharding in these cases. Our Open Source infima library doesn't quite include direct support for this, but you can get a sense of how we put it all together:

https://github.com/awslabs/route53-infima

Cool, thanks!

Colm and Marc frequently post great threads about distributed system design on Twitter. Definitely worth a follow: https://twitter.com/mndoci/status/1202641319643869184

And on a personal note, I’m astounded and so lucky that I get to work with these amazing engineers!

I've worked in AWS for five years. I get paid, but the kind of knowledge in these pieces are by far the most valuable thing I've gotten from working there. I've been in it. I've used these principals. I've seen what happens to drive these recommendations first-hand. I've worked to design and build systems that follow and support these guidelines and principals.

Amazingly little to no use of images in the articles. Turning the 'an image is worth a thousand words' on its head. Seems to work well though. Examples: https://aws.amazon.com/builders-library/avoiding-insurmounta..., https://aws.amazon.com/builders-library/caching-challenges-a...

Good point. We should use more images. I’ll think about where we can add some for clarity! Now that we’ve done a bunch of talks at reinvent in these topics, we should have some that we can incorporate easily.

Disclaimer: ex-AWS

Wowzer. Amazon, and in particular AWS, sits on a treasure trove of hard-earned lessons abt designing, building, operating, maintaining systems at scale. I'm glad they have began to share some of that know-how with the external world.

I wrote abt longing for AWS to do this some 6 months back [0] as I dearly missed access to those docs and videos that helped me understand so much abt the underpinnings of some of the most staunchest challenges dealt with by behemoth-esque services like S3, EC2, Redshift, DynamoDB, Kinesis, LB, Lambda, Edge, Security to name but a few.

I must point out though, re:invent does have a ARC track [1] which gives glimpses into AWS' design principles.

There's so much they could still share (for instance, abt how they track and run their operations, how they deal with service outages, how they do change management, how they continuously monitor and improve the weaker and slower parts of the systems, how they gear up for launches, how they deal with issues facing larger customers, how they go abt their design process, how they refine their internal tools and processes, the internal tech talks, the review meetings esp the one run by Charlie Bell...), and I'm glad they've committed to be doing just that.

It'd be fun comparing literature coming out of Amazon to what's published by Google et al.

[0] https://news.ycombinator.com/item?id=20251589

[1] https://www.youtube.com/playlist?list=PLhr1KZpdzukfzSjpgH4E5...

why do you keep typing 'abt' instead of 'about'? what are you planning to do with all the time saved?

Isn't abt [0] a generally accepted abbrv for about [1]?

[0] https://www.dictionary.com/browse/abt-?s=t

[1] https://www.abbreviations.com/ABT

It's understandable, but very rare in most text. It's the sort of thing you'd expect in a telegram, not a modern forum comment.

Especially when they take the time to write "management", instead of "mgmt".

> what are you planning to do with all the time saved?

This made me laugh hilariously.

But overall the parent was a very helpful comment, to get this perspective from an ex-Amazon-er

This is peak HN right here.

Thanks. I thought abt was some technology I hadn't heard of.

The Charlie Bell ops review meeting (and the various 'make sure you don't embarrass us in front of Charlie' pre-review meetings) is especially critical. It's not just about having the techniques for doing ops well, it's about leadership prioritizing it through both words and actions.

Does anyone know why Amazon engineers always seem to write so well? Any time I read technical content from Amazon, I am struck by how clear and concise it is.

Amazon has a very strong written culture (search for 1 pagers and 6 pagers). To be effective at higher levels in any role you need to be good at it.

Any tips on how to get better at this?

Write more. Read more. Seriously. It just takes practice.

https://ceklog.kindel.com/2018/04/24/be-a-great-reader/

Of all the very specific suggestions and recommendations, this really is the best one. There is really simple technique for getting better at anything: do more of it.

I wonder if this is simply because human brains are just create specialized neural pathways for special tasks that occur more frequently. So it takes a lot of effort for someone who has never driven a car to drive but after they learn driving it’s automatic (same for language and any other skill)

My advice is to take a step back and consider what you want the reader to take away from what you've written. Then make sure it flows, so as you introduce one idea, make sure what you wrote prior is enough in terms of pre-requisites,

One way would be to read a lot of the type of writing you want to emulate.

But you'll get the best results by putting a lot of directed and specific practice into it. Ben Franklin wrote about his method in his autobiography. He would try to reproduce passages he admired from memory. Then he would check his work and repeat until he reproduced it verbatim. Then he would repeat but try to improve it.

Write more and find ways to get feedback on your writing. It's important to get outside input to help you see the things that don't work and experienced readers/editors/friends/coworkers can often provide suggestions on how you can fix it.

This. and: It really helps if you work in a culture that values critical feedback on thinking and writing. Too many readers will pass on an opportunity to offer critical feedback, either because they don’t want to offend or because they didn’t read it carefully and critically enough. Having someone read a doc and tell you it’s good feels good, but in general doesn’t help you improve (and is actively harmful if the document is mediocre).

One way to encourage this helpful criticism is for authors to accept feedback gracefully and have a policy to ask questions only for clarification of the feedback (to understand the feedback better) and never to engage in the debate response of explaining why what they wrote is good enough.

(The minute you debate them on the first of seven points they have, they’re going to deprive you of much of the value of their feedback on the next six because you’re seeming defensive at worst or exhausting at the least.)

Suggestions are great. Whats better is just taking anything that people don't understand as "area for improvement". Then write it and see it it's better. Iterate. You'll learn your own style.

Another possibility to take a writing class at a local community college. A lot of it is common writing strategies like how to organize thoughts (spatially, conceptually, temporally, etc). And as others say, practice. The classroom is nice because it forces one to practice and gets feedback from the instructor or writing lab (most schools I’ve been to have them)

Start with the basics. Read a business writing book, articles, or take a course on it. Even a one-hour course will teach you something new. Beyond that, some things that help me address my weak spots:

1. Make sure your first paragraph or executive summary answers the questions: Why does this doc exist? Who is the expected audience? What actions or decisions does the author expect from the audience after reading the doc?

2. Run things through an analyzer like Hemingway[1]. It'll point out obvious things to fix.

3. Do a reverse-thesaurus editing pass. Remove adjectives and flowery language where possible. Challenge yourself to lower the reading level. Even if your audience has PhDs they'll read and comprehend simple language faster.

4. Do an editing pass for missing numbers. Vague language, unsupported assertions, and missing quantities make arguments easy to refute. Look for words like "many", "most", "a lot", "major" "severe" "large" and replace them with hard numbers where possible.

5. Do an editing pass for "So, what?" and remove anything that isn't necessary to support your core argument or purpose. Assume your audience is smart but has very little time. Too much detail will make them start to skim and miss things. Appendixes are your friend here. Leave links to appendixes for readers that have questions or want more detail.

6. Nothing beats a human reviewer. Professional writers have editors too.

[1] http://www.hemingwayapp.com/

Write A Lot

When I started at amazon I was encouraged to pitch a project or feature once a month. More importantly I was encourage to write up things I disagreed with or try to sell stuff that was very half assed, and work on getting it to a real idea.

As I did this I got better and better. I can write up 3-5 page design docs in about 1-2 hours now and these can bootstrap others into your thinking very easy. I've found that they also just ground an argument when everyone is arguing about totally different things. Even if they are now all arguing AGAINST your idea, you've made progress.

It's just a muscle you have to exercise. And iterate with feedback. Learn from where people have trouble understanding your document. If they don't understand it, take it as feedback that you are not clear enough.

TLDR Write Write Write

One of the authors here. I agree with all the folks’ suggestions here! When I was in college, I took extra writing courses beyond what was required for an engineering degree (they counted toward general humanities credits I think). People always told me how important communication and writing was in engineering. Not sure the courses are what did it for me entirely, but it certainly taught me concise “technical communication”, as well as story telling, and forced me to practice! A huge key to good, clear writing is reviewers. I ended up essentially rewriting some of the articles multiple times based on feedback. And we all had some great editorial help on these articles to give the articles that final polish.

Amazon really emphasizes hiring and developing writing ability. I’m convinced it’s a major reason they’ve been so successful.

(I worked there 2006-2016)

Can you talk at all about what they do to develop writing skill?

> I'm convinced it's a major reason they've been so successful.

I'd be interested to hear why you think this.

One guess I've got is that a high standard for writing is a filter for bad ideas - if you can't justify your idea in a tight, well-written document, then it's probably not worth trying out.

They have classes and workshops for employees who want to get better at writing, but primarily they just expect you to get better with practice. I wrote a few one pagers every month as an engineer, and managers produced an order of magnitude more.

You've got it more or less. The way I say it is that bad ideas can't hide in writing. You can fool yourself and others with a powerpoint or a spoken argument. But clear writing demands clear thinking. Poorly constructed arguments are (usually) obvious as such when put into writing.

Writing is, indeed an essential part of Amazon's culture. I also am convinced that the writing culture contributed to the company's success. It's not just PM or SDMs. Engineers at every level are expected to write well. Most meetings often start with reading the time, even up to 30-45 mins. It does work and forces people to think as there is no difference between writing clearly and thinking clearly. I think everyone recognizes that and uses the culture to their advantage. As an SDE, I just love it.

Reading this snippet in the article about leader election in distributed systems: "Each item of data still belongs to a single leader, but the whole system contains many leaders.". It could not be a clearer statement of Conway's law[0]. Orgs ship their org chart. The technical architecture at AWS absolutely reflects the org chart and business organisation of Amazon generally.

[0] https://www.techopedia.com/definition/31927/conways-law

Lot of amazon content in the past week or two. What's going on? did they have a keynote or something?

AWS re:Invent (https://reinvent.awsevents.com)

It's great to see some of the stuff that I used while at Amazon becoming more publicly available. Good documents, these.

Is that something new? Or they just compiled all their tutorials in one place?

This just got released. It’s a living source authored by senior engineers at Amazon and covers some of the best practices that we use at Amazon.

I'm really glad this is public now, really good experience-driven (often straight from major post-mortems) recommendations here.

Some real gems there.

yeah this is great stuff!

Lists don't make such great HN posts, because the discussion inevitably becomes about the lowest common denominator of the items on the list. The interesting material is in the specific items on the list, but these never escape the gravitational pull of the generic topics (in this case Amazon and AWS). It's usually better to pick the most interesting thing on the list and submit that instead.

https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

Normally I would agree with you, but the existence of the library was the most significant announcement of today's re:invent keynote. So the existence of the list, and the fact that the list will keep growing, is the news here.

One option might be to change the URL on this submission to the blog post about the list instead:

https://aws.amazon.com/blogs/aws/check-out-the-amazon-builde...

That's a fair point. Thanks for the correction!

Reply on news.ycombinator.com