redredrobot

There has not been an AI winter in at least a decade, arguably more.

It's a good reference because people are familiar with GPT-3. The paper mostly compares Chinchilla to LaMDA, Jurassic, Gopher, MT-NLG, and GPT-3. In the broader tech industry and even to a certain extent within the AI field, GPT-3 is the only one that most people know by name.

It outperforms the Gopher model

Lol when I said that at an org-level all-hands everyone laughed, but the need for improved PR is very real.

But it’s important to remember that there are a ton of extremely happy people at Amazon (or at least AWS, where I worked) they just aren’t as vocal online - management, scope and culture were incredible in my experience (it’s a huge company where managers have a lot of flexibility, YMMV). And some of the biggest day-to-day issues you hear about like crappy offices or excessive frugality were a thing of the past when I was there. And in terms of PIP, I only had one experience with it - having a mechanism for identifying and getting rid of the people who are a drain on everyone else was very nice.

At least when the approach works, I think the goal is to not have to have the teams aligned, instead giving them independent mandates that lead to the emergent behavior being what the business needs.

Yeah, that's a fair point.

Well put, I think this is very true.

But I don't really agree. If you are a small team solving a small part of a larger problem and you have done a good job of defining your external interfaces, you can make decisions about how to solve your subproblem quickly and independently. Also, process/culture decisions (e.g. code style, automated tooling, code review standards, update cadence, etc) become easier because you only need to come up with a solution that makes ~10 people happy instead of a solution that serves the needs of 100+ people.

I worked at Amazon on two-pizza teams. I never thought the two pizza team was about reducing communication, but more allowing faster independent decision making. Once you work on something that is big and complicated enough that it requires dozens of engineers, there HAS to be communication and coordination. Hopefully you have good managers (and high level ICs) because they are supposed to do a lot of the coordination so that it is easy for the engineering team to focus on engineering. A goal of independent two-pizza teams is to have the independent teams define how their individual pieces will interface and then the small teams can independently decide on implementation details.

The two-pizza team concept is also about having a heuristic that tells you when to start looking into how to divide responsibilities between independent teams.

I don't really get what this article is proposing. Never work on projects complicated enough to require lots of engineers and thus coordination? It is not realistic to deliver the complex projects that big companies need to do with a single small team doing everything. It will just never get delivered.

I guess my point is more that as a huge tech company that wants to hire many very good ML researchers, a culture of publishing is necessary

Not to take away from the people who worked hard to shift the culture (which is always hard at a large company), but I saw the same shift happen at Amazon and it came down to a simple fact - it's very hard to recruit and keep the best talent if you don't publish. You don't build ML reputation and the best researchers often don't want to work somewhere where they can't add papers to their resume.

I worked in an area where it is really hard to figure out exactly what workloads were being run and where it would have been extremely useful to know even basic things like CPU utilization patterns, network throughput patterns, etc for a specific customer.

We had access to absolutely none of that information. We flew blind, relying entirely on the fact that we gave our customers enough hand-holding support that they would willingly volunteer information about their workloads so we could help them optimize it/save money.

No one even attempted to get more detailed customer information AFAIK because it would have been extremely against company culture. That isn't Earning Trust or having Customer Obsession. The idea of reading data in someone's S3 bucket or inspecting what is happening inside of someone's EC2 instance in any way was unthinkable. Amazon is huge and imperfect, but from what I saw AWS takes data privacy extremely seriously.

What do you mean? Prior to COVID I thought Bezos had his full attention on Blue Origin

Actually if you pay a little attention to Bezos, it's clear that his focus has been outer space at least since he was a college student. IMO Amazon was an opportunity that Bezos saw while Blue Origin is the fulfillment of his lifelong dream.

> Amazon doesn’t invest in greenfield projects that are not customer facing

I don't think this is very true, they are just very good at turning around and selling the internal tech.

The underlying technology for AWS was developed internally and it was not customer facing at first.

Similarly if you look at their internal data platforms, they have invested heavily even though it's not customer facing (yet, because eventually they will start selling services inspired by the internal systems via AWS).

I remember watching a presentation on how they invested large amounts of money into prototyping FC robotics technologies.

If you look at AWS Kinesis or EventBridge, the technologies were developed for internal usage before they were sold. There is also considerable investment into distributed consensus technologies that hasn't been turned directly into products (yet).

You're kind of right in that Amazon doesn't spend much time working on greenfield projects that don't have any forseeable business value (in contrast to say Google), but that is in many way a product of their organizational structure where two-pizza teams are supposed to do 'research' as part of their product work rather than having a pure R&D division.

I found that weird. On-call is so central to the technical identity of Amazon that I'm surprised you could get through the hiring process without knowing that you'd be on-call.

Interesting. You're probably right, the experience is different for new grads where the lump sum is really useful.

> I mean in the sense that being customer obsessed should mean you do not evil

I think there is a real tension between customer obsession and 'do no evil'. Serving the individual needs of consumers can come at the expense of the public good. I think example of that are the amount of waste that Amazon packages generate by shipping as fast as possible and the amount of environmental damage AWS can quietly cause via their datacenters.

How so? Google is facing employee issues and protests because they sold the 'do no evil' ideal, but the employees don't think some of their projects follow that. I think you would struggle to find an equivalent portion of Amazon employees who think Amazon isn't customer obsessed.

I appreciate this being shared, but I have some qualms.

I think talking about the Leadership Principles as a 'typical corporate motto' is unfair. Amazon lives their leadership principles far more than the other companies I've seen. They don't include things they don't actually care about, like most corporate mottos do (you'll notice the lack of 'we care about our employees' or 'do no evil' LPs).

At least in my experience, you only need to pay back Amazon proportional to (2 years minus duration employed) so it really isn't that bad (maybe that wasn't the case when this was written).

Maybe it's changed, but the signing bonuses I saw were essentially increased salary until your stock starts vesting seriously. It gets paid out and vests every paycheck so there isn't anything you need to pay back.

The Charlie Bell ops review meeting (and the various 'make sure you don't embarrass us in front of Charlie' pre-review meetings) is especially critical. It's not just about having the techniques for doing ops well, it's about leadership prioritizing it through both words and actions.

I don't know. I moved from AWS to a <10 person startup and it has a very similar feel.

Making it hard to cross regions is deliberate - I would guess it's a learning from s3 being designed as a global service. And I think if you compared global outages across cloud providers, you would see that strong region isolation has substantial benefits.

> a GPU vm with a local SSD on AWS, does not exist

I'm pretty sure that's not true. I believe you can get V100s or T4s with local NVMe SSDs.

I don't think your statements about (a) where cloud margins come from and (b) Lambda vs VM cost are correct.

My understanding was that market share is: AWS > Azure > GCP > Alibaba

That's not seriously the real engineering portmortem is it? That looks more like a 'Resolved the issue' update - it is way too shallow and vague.

If this was sent out at AWS as a COE (postmortem), it would be ripped apart - it is not going to satisfy anyone reading it that they should have confidence this class of failure isn't going to happen again. It looks like they haven't even identified the root cause(s) of the failure...