Hacker News Reader

Comment by Sevii | Hacker News Reader

I appreciate that they have a programming philosophy that they want people at the company to adopt. A common problem I see at companies that don't have onboarding is that people join the team with assumptions from previous jobs but you never level set them with the company. So 12 months down the line the new guy wants to change the process and you have to repeat the same discussions about what agile means for the nth time.

Amazon does a good job of training new hire on the 'Amazon way'. Amazon does 6 pagers, they do design docs. Amazon does SOA. Amazon does not use relational databases. Everything has an API. Because of the 'Amazon way' and the training they do new team members understand at least some of the context and expectations.

Is it the best way? Probably not but no one knows what the best way is anyway. At least they have a way. Saves a lot of effort compared to every new hire relitigating the process and architecture.

View on HN

> Amazon does a good job of training new hire on the 'Amazon way'. Amazon does 6 pagers, they do design docs. Amazon does SOA. Amazon does not use relational databases. Everything has an API. Because of the 'Amazon way' and the training they do new team members understand at least some of the context and expectations.

As a counterpoint, a huge part of Amazon's culture (or at least, AWS's) in my experience was the emphasis on operations and the fact that they didn't have any separation between SREs/on-call engineers from the people who implement their services, and at least for me as someone who had never been on-call before in any meaningful capacity (due to my previous job working on a libraries rather than services), the training for it was basically non-existent on the two teams I spent time on. The "training" I did receive essentially consisted of being put on the rotation once to shadow, where I was able to sort of see what the actual on-call person did but didn't really have any explanation for how to know how to do them other than being told to read the runbooks, which were not really written in a way that was easy to understand for me as someone who was so new to learning all of the internal AWS tooling and ops in general. The next time I came up on the rotation, I was expected to be able to manage on my own, which essentially meant that literally no matter what occurred, I ended up having to escalate because I wasn't knowledgeable enough to fix literally anything within a timeframe that would have been reasonable.

Which is the only way to learn tbh, you can receive as much positive reinforcement imaginable, nothing prepares you for a large scale incident like living through one, building the connections you need to solve it, getting the shame of your life, and losing sleep over your failure.

Nothing teaches you to swim like being thrown in the middle of the atlantic.

I'm not really sure what you mean by "positive reinforcement", but I don't think it's possible to disagree more with this sentiment. "building the connections you need to solve it, getting the shame of your life, and losing sleep over your failure" isn't a strategy for teaching for something; it's a coping mechanism for someone trying to brute force their way through something that they weren't adequately trained for.

Most people seem to think it's fine for companies to offload the entirety of the burden of learning to individual employees, and maybe I'm an outlier in this regard, but to me, this seems more like a cop out to avoid trying to actually solve the problem at the cost of the employee's emotional health. I'm not surprised that companies default to this, but it's also not surprising that burnout is so common in our industry when this is considered the "best" or "only" way to do things.

> they didn't have any separation between SREs/on-call engineers from the people who implement their services

I.e., treat DevOps as a way of working, rather than a role meaning something akin to "Ops person who knows terraform, or k8s, or Ansible etc".

My team didn't add new members to the oncall rotation for about 6 months to ameliorate this issue. But starting oncall at first is rough and even with months of context on our systems people usually take a few rotations before they really figure it out. We expected new members to have to escalate.

I don't think this really ameliorates the issue much; it just pushes the problem down the line. IMO this is a big part of why people transfer internally to different teams so much at Amazon, and that masks the problem even further. If by six months of on-call rotation you expect someone to be self-sufficient, but they only start after six months on the team, they'll have been on the team a year by that point, and people either transferring or leaving the company after a year on average isn't going to be immediately obvious as a problem, but if people start a few weeks in, you're going to have 5-6 months of noticing that there are issues when that person is on-call, and if that happens more than a few times, the trend will be noticeable.

> Amazon does not use relational databases

This is false, at least in my very thin exposure to the company: I interviewed for a team last year which was maintaining EC2 SSH keys using MySQL.

I watched an interesting talk from the RDS team about how they dogfood RDS the other day https://www.pgevents.ca/events/pgconfdev2024/schedule/sessio...

If it was using MyISAM relations are in question.

SQL, and therefore MySQL by extension, isn't relational.

He typed smugly, confident that everyone reading this will appreciate his pedantry that totally contributed to the conversation

A lot to unpack in this comment.

1. Wherein do you find the smugness? It does not speak to any person, let alone the first person.

2. What would give the impression that comments on the internet are written for others? Carly Simon once recorded a popular song about this type of falsehood.

3. It remains that SQL isn't relational. That is why it "won", after all. Relations are too complex for the layman to understand. Tables are much more familiar to the people in charge and arguably a better model for most business problems.

> Amazon does not use relational databases

Huh?

Friends say they typically use Dynamo and that using a relational database requires approval from a vp (because of scaling concerns).

Related: Amazon kicked out Oracle from the company.

Somewhere along the same timeline, the operational recommendation for teams was to not use relational databases.

https://www.theregister.com/2019/10/16/amazon_ditches_oracle...

Kicking out Oracle might have to do with factors outside the usage of relational databases...

Relational databases are not the preferred storage mechanism at Amazon. If a team wants to use an OLTP relational database it’s possible that it will be a decision they will need to defend at any kind of design review.

Of course there are relational databases running OLTP workloads, but it’s far away from the norm. There was a program a while ago to shift many RDBMS systems onto something else.

So they do joins in code rather than SQL? Wouldn't that risk hiding scaling problems?

It can but it's usually more obvious what's happening with code and how to fix it. Amazon wants you to think about the scaling issues while building as they don't want to lose the area under the customer curve on the far right.

The theory is that with rdbms you have a magical box that scales vertically until it doesn't. And when it doesn't all you can do is scale back the customers until you fix it with sharding or a re-architecture. Basically you tend to hang yourself with indexes and transactions. Also generally when an RDBMS fails it fails down to like 30% throughput.

I recently finished a contract at a company who has gone full on dynamo with the idea that if we have slow queries and dynamo is good for Amazon, then it's good for us too. I've ran some explain on the queries causing issues and of course those queries didn't leverage indexes like they thought ....

How did you run explain queries on DynamoDB? Or may be you mean something different and I misunderstood you?

I ran explain on the original mysql implementation. The basis of the migration project to dynamo was mysql couldn't cope with our scale but that was bullshit

That doesn't make sense, you have to specify the index when you're using Dynamo.

You can do scan operations and if they use PartiQL it hides whether you are using indexes.

I usually have an explicit DENY for dynamodb:Scan for the IAM role used to access the DDB table

Look up design patterns in DynamoDB. If you know your access patterns and you often do with well defined microservices. You don’t need to do joins.

Amazon Retail has multiple systems to allow you to basically use SQL across databases (eg Datapath).

I’m not sure about AWS.

Reply on news.ycombinator.com