Amazon's empire rests on its low-key approach to AI | Hacker News Reader

Amazon's empire rests on its low-key approach to AI

economist.com

157 pointsseagullz92 comments

> AMAZON’S SIX-PAGE memos are famous. Executives must write one every year, laying out their business plan. Less well known is that these missives must always answer one question in particular: how are you planning to use machine learning? Responses like “not much” are, according to Amazon managers, discouraged.

I think company culture & talent make a big difference here. At most companies this would run the risk of poorly shoe-horning ML into a bunch of applications and making everything needlessly complex & expensive without seeing improved results (same goes for blockchain).

Can anyone at amazon comment on whether this actually works?

Edit: the ai part specifically.

From what I have seen 6-pager process and more importantly the clear writing and efficient meetings that it fosters are pretty helpful. It's important to note that a 6-pager contains 6 pages of expected reading and can have as much additional appendix material as you want. I've seen "6-pager" documents that were dozens of pages long, and have heard possibly apocryphal stories of 6-pagers hundreds of pages long for major projects. If the question was specifically about AI being a required component, idk. But a VP at Amazon presides over a part of the business so large that it is implausible that there aren't useful applications for AI in their domain.

The ideal of the 6-pager is magnificent. The reality of the 6-pager in 2019 is that it has become boilerplate where style overwhelms substance. Similar things can be said of their OP1 and OP3 planning processes that are becoming increasingly cargo cult IMO. And that arises naturally from a static game whose most experienced players have become better at funding projects than they are at funding better projects.

Though very late to the party (modulo Alexa), Amazon has eagerly embraced AI and machine learning since 2015, but it lacks the leadership for formulating clear targets (see AutoML and AlphaGo Zero as examples of clear targets IMO).

A recent article in _The Information_ made the claim that their entire AI organization is making <$20M annually.

That is consistent with what I saw there: an effectively infinite number of codemonkeys throwing unprocessed data at randomly chosen AI algorithms downloaded off of github and hoping for the best. It's an interesting experiment, but it seems to be about as efficient as a monkey throwing darts to pick stocks (which is surprisingly better than many biased investment advisors admittedly). Time will tell, no?

https://www.theinformation.com/articles/demand-soaring-for-a...?

They need a 6-pager czar who has the power to send back to editing any paper that's not readable enough.

That just pushes the problem one level down on the stack, no? Or who sends back the work of the 6-pager Czar? IMO they need to refresh the process, and refresh it frequently. I don't see them doing that though.

Similar things have happened with the well-intentioned "bar-raiser" process as well IMO. I had bar-raisers in interview loops for my team who seemed more intent in blocking potential competitors to their niche than in hiring the best people, which was the whole point of said "bar-raising."

No, distributed responsibility is qualitatively different from a single responsible party.

They just need an AI to review for sub-optimal 6-page reports. First cut can be built off of mechanical turk.

> The reality of the 6-pager in 2019 is that it has become boilerplate where style overwhelms substance.

I think this varies by team. In my experience thus far, I've been receiving lots of feedback -- often actionable -- to improve my own narratives.

If your reviewers aren't pushing back on you when your paper lacks the required substance, that definitely needs to be addressed.

> OP1 and OP3 planning processes that are becoming increasingly cargo cult IMO

Businesses need operating plans to determine prioritization and funding. What's the problem with them, exactly?

In my case, the project we wrote an OP1 for was judged to be too technically complex such that anyone capable of executing on it would be a "flight risk" capable of commanding higher pay elsewhere.

Therefore they decided to destaff the effort and wait for me to prove them right rather than give me a couple engineers and enough rope to hang myself (which I wouldn't have most likely given my track record that established me as a "flight risk" apparently).

Can we say self-fulfilling prophesy?

As an Amazonian far flung from AWS, (I am in ground level fulfillment) I have seen some incredible internal AZ programs that utilize ML in unexpected areas to save the company lots of money. So I'm going with yes.

The article overstates the use of ML at least in fulfillment we use a lot of math models (eg xpress solver) and optimization algorithms (traveling salesmen type problems) but not so much in the ML space.

My team does ML in Fulfillment. Search my username on phone tool to figure out my team.

Usually optimization takes forecasts as part of inputs. It is actually a lot like reinforcement learning, where the sales forecasts are part of the environment. This is standard process for supply-chain management, not just retail industry or at Amazon.

I have never worked for Amazon so I'm just speculating but if you have a culture of being data driven, specifically in the ways you optimize things then I'm guessing you'll probably be pretty successful and it's not a giant leap to plugging that data in to some ML models or something. To plug that stuff in the some ML models requires an entire culture of data collection and some thought as to how you'd improve or benefit from it; it's sort of like the "no brown M&Ms" rider that Van Hallen used to have.

There is also the story of Amazon mandating that all internal systems communicate with each other through APIs. I think leadership at Amazon has good technical knowledge so they can develop these strategies and see them through.

In my company nobody with a "C*O" title (including CIO and CTO) has any clue about tech so they are very susceptible to snake oil salesmen telling them things like "we must do ML". So they tell some random person to "implement AI" but nobody has any clear objectives or can make a judgement whether things go in the right direction. In most cases these things end up as expensive disasters. I am waiting for the "Blockchain Excellence Initiative" with a lot of presentations and shiny newsletters.

How can someone become a CTO without having any clue about tech?

In a big enough company where software isn't the product that just means they know the tech that applies to that company. The CIO is probably a fearful late career IT politician and the CTO could have spent 40 years progressively learning more about washing machines or planes.

That's pretty much it. There is also a widespread belief that software development is "IT" which means that an initiative to implement cloud systems gets led by people who have experience with desktop deployments or firewall setup. There is a lot of politics around budgets so the IT department won't let go of that power.

I read in Werners Vogels blog (amazon CTO), that there are four types of CTOs [1] based on Berray and Sampath paper. [2]

* Infrastructure Manager

* Technology Visionary and Operations Manager

* External Facing Technologist

* Big Thinker

[1] https://www.allthingsdistributed.com/2007/07/the_different_c...

[2] http://www.brixtonspa.com/Career/The_Role_of_the_CTO_4Models...

That list is not exhaustive. One might also encounter "Peter Principle Beneficiary".

Another bit that not many people know is that CTOs at big non-tech corps can just pay a consultant to literally tell them what to do (most of the big banks do this) and consulting firms make a ton of money providing this service (namely places like Deloitte etc).

I have the feeling that's how IT is done at my company. They have a huge number of outsourced contractors in India and the US and not very many people who really understand technology and can move things forward.

“IT politician” is a great phrase, I will be stealing that - thanks!

Or a security chief without any clue about tech (let alone security):

https://it.slashdot.org/story/17/09/15/1910200/equifax-ceo-h...

It's still relatively common for people in IT in general and security specifically to have no formal background in it, so citing what someone did ages ago in university isn't all that convincing to establish lack of "clue about tech".

I have worked with a lot of CTOs or similar. Sometimes CTO is a title that can approve large budgets and an effective CTO is actually their direct report.

Being a CTO is less about technology and more about understanding the global technical landscape, accounting and legal implications to large budget decision making. If I, for example, choose Microsoft as a vendor will that impact our Pacific market and if so what is the cost benefit? These are not trivial choices. To an extent a lot of the job is risk mitigation, reducing liability for the company.

A good analogy would be a CEO of a shipping company not knowing how to sail. They will know costs of the voyage, costs associated with the voyage, etc. but not the day to day of how a ship operates. And they don’t need to. If anyone has worked with a micromanager they’d appreciate this.

Sure but sometimes this lack of knowledge will cripple a company. I've had a VP(who had a role similar to the cto for the department) who picked what time series database we should use based on this technology being the best one from a business standpoint. Unfortunately the technology was so bad from a technical standpoint that every single project that used it had it's budget and timeline explode 4-6x. This seems like an exaggeration but it isn't. One literally wrote a working application in 6 months then was mandated to move to the new technology (basically rewriting the application) and they spent 2 years rewriting while still leverage a lot of the code they already wrote.

And I feel like 100s of these decision are made everyday by executives who oversee but don't grok software.

Out of more than idle curiosity, what was the TSDB?

I would say but it's a small proprietary one that only the company I was working for used. So mentioning it doesn't help any devs avoid a mistake and outs the project.

My bet is opentsdb?

It's more about the degree than practical reasons in many cases.

Business analysts, attorneys, finance people and IT turned PM/administrative all manage to rise to high technical positions due to business savvy and political ability.

I worked in a place with a CIO who didn’t know how to operate a TV. If you aren’t a tech company, it’s often a liability to have too much tech domain knowledge.

> I am waiting for the "Blockchain Excellence Initiative" with a lot of presentations and shiny newsletters.

That's actually a really good business plan.

There are already plenty of them...

https://www.google.com/search?q=blockchain+center+of+excelle...

So far most of things I have seen that have the name “Excellence” are bullshit, produce a lot of hot air.

Interesting thought, thanks for sharing

I have a theory that much of the success of "internet" companies is less due to disruptive business models, and more because they can software.

For sure, but it's sometimes a mix of both.

At most companies this would run the risk of poorly shoe-horning ML into a bunch of applications and making everything needlessly complex & expensive without seeing improved results (same goes for blockchain).

Sure, and in Amazon, you get a few tweaks that get the company a few bucks. It suspect the article wildly overstates the value of ML to Amazon.

Amazon's success seems to rest on: A. Attracting enough capital to get huge and stay huge with generally low margins. B. Being willing to slide into related businesses for extra-profits. C. Not jumping into one or another dumb trendy investments that would burn up all the capital they get from A & B.

Well, the memos might help B & C, for all I know.

Ten years ago, Amazon and Ebay split the ecommerce/storefront world. Ebay never managed to make themselves fully consumer friendly with hard guarantees and Amazon did. The auction-based, at-your-own-risk world is inherently limited and Ebay never escaped that - I wish they had 'cause then Amazon would have had real competition.

Wal-mart is real competition.

Amazon now is something like a fusion of Ebay and Walmart. Either of those enterprises (or Target or etc) could move into the full space that Amazon operates in, it seems like.

Edit: I suspect that what keeps an enterprise like Walmart or Costco from jumping into full competition with Amazon is that each of these enterprises wants to fish some high profit zone of the eComerce ocean while Amazon wants to be the ocean.

I was curious how the Amazon/eBay split looks revenue-wise.... Not exactly apples-to-apples but in 2018 Amazon was $180B in net sales vs eBay $95B GMV.

So Amazon is more than double but maybe not 10x? Amazon GMV is supposedly in the $250B range but they don't report it. And it's obviously hard to compare such different business models.

Not exactly apples-to-apples but in 2018 Amazon was $180B in net sales vs eBay $95B GMV.

So you're comparing Amazon sales to eBay market value (total stock price). Yeah, it's completely meaningless comparison [Edit: I was wrong, it's Amazon sales versus eBay's gross merchandise value but it's still entirely off as a comparison].

Btw, my quickly googled references show eBay's net sales as 10.75B and Amazon's at $141.92B, which is indeed a more than 10x relations (when I think all other statistics show Amazon an order of magnitude ahead, not that I initially made any strong assertions about this but anyone looking has notice Ebay's star fading).

[1]https://amigobulls.com/stocks/EBAY/income-statement/annual

[2] https://www.digitalcommerce360.com/article/amazon-sales/

GMV is Gross Merchandise Value, not stock price.

https://en.wikipedia.org/wiki/Gross_merchandise_volume

eBay states there's here: https://ebayadvertising.com

The question is what is the GMV for Amazon's entire marketplace vs. their first-party Sales.

Okay, now I know, but the fact remains that Amazon is actually much large than Ebay by all apples-to-apples measures.

My concern with the net sales comparison is that I think for eBay that only includes their fee, not the actual retail price of the goods, whereas for Amazon it is the entire price of goods sold. That to me seemed a much more apples-to-oranges comparison which is why I didn't cite it. I have no idea what's actually true. I would be interested if someone wanted to sort out the truth there but I hit my limit for how much more research I want to do.

Some specialty items should only be purchased on Ebay. You get the same products built in the same Chinese factories, without the markup for marketing and salaries required to involve USA middlemen. Inexpensive manufactured items of more general interest can be found lots of places: AMZN, WLMT, Harbor Freight, etc. Ebay is filling a niche for the less common stuff, though.

Feel like you've massively glossed over Amazon's strategy. From working there, data-driven decision making and automation permeated throughout the whole company culture.

For me, this was evident from the fact that quite literally everything was automated:

Physical SKU distribution was allocated by an algorithm, optimizing for space density and picker path length. Pickers carried trackers to enable the data to constantly be gathered and algo performance reassessed.

Prices were automated in real-time

Inventory management and purchasing was automated and humans were there to override when necessary.

Kiva was purchased and applied to implement an inventory store that can self-reorganize in real-time

At least in my mind, this goes far beyond the token "we do A/B testing on UX" that other webshops do.

Worth also mentioning that Amazon did collaborative filtering on book recommendation way before it was trendy to do...

When I squint, it feels like Bezos had some image of some giant razor thin machine (quite literally) selling commodity goods, constantly capitalizing on competitors' inefficiencies. I can't help but feel this was inspired by his time spent at DE Shaw...

I was discussing Machine Learning in particular. The wider use of automation is certainly something goes along with building at scale, so yeah that deserves mention.

I think a lot of people just don't realize that Machine Learning can be as simple as a linear regression with 2 variables. The hype and myth around ML are such that some employers just assume that they need to hire rockstar data scientists to do ML but in reality, a junior level data analyst or even a non-technical employee can do simple ML model in Excel spreadsheet assuming he/she has some background in Stats 101.

Yes but that is the easy part. The hard part is knowing what can be solved with ML, gathering the training data, validating the model and then explaining what the model does.

Acually fitting the model is not the hard part.

But yes, not all ML is deep learning and most ML problems should NOT use deep learning.

>>The hard part is knowing what can be solved with ML

I’d say THIS is the hardest part by far. It requires a very rare mix of business domain expertise and technical knowledge with regards to ML/DL.

> Acually fitting the model is not the hard part.

Ideally you shouldn’t be fitting the model yourself. The idea behind ML is to use brute force computation.

If you’re fitting the model by hand, you’re a Statistician.

Statisticians do not fit models by hand, they "fit" models in the same way machine learning pratictioners "train" models. "Fitting" in statistics lingo is completely synonymous with "training" in ML lingo. Statisticians were the original ML practitioners.

I will second this. I use the two words interchangeably. I would probably use the word fit when talking about liner regression and train when talking about a more complex ML model that requires more computation. I really don't remember the last time I fitted a regression model by hand, it would have probably been in a college exam.

And yet I still get emails from Amazon trying to sell me products I just bought.

Bought a hygrometer earlier in the week, delivered yesterday, email today about hygrometers.

Maybe they have data showing that converts well.

My feeling: amazon is great in applying ML in the back, e.g. warehouse replenishments, but not so great in the customer facing applications. Only my view from outside

Actually, they do! This is a frequent misunderstanding, but buyer's remorse is a common thing and a lot of folks return the items that they've bought because they're unsatisfied with it. Even if it is 5% of all customers, the numbers will add up really fast for Amazon.

It's quite clever if you look at it from an aggregate perspective. They're casting a wider, second net to catch the fish who slip through.

I bought a hygrometer to test on one plant, works well, going to buy some more (actual situation). I may not need more than one trimmer but with high probability Amazons algorithms know that some products are bought more than once.

This year, our internal ML conference ran out of tickets within 90 minutes of opening registration. There is a whole helluva lot of ML research and applications going on.

Tell me about it!

- sincerely, someone that didn’t register in time

https://outline.com/34PN5W

I ran Kindle software Product Management for 3 years. Left in 2014

AMA about the memo process or OP1/OP2 process.

Did you ever feel like it was a waste of time or do you feel it genuinely added value to the company to engage in that exercise

Did you ever feel like it was a waste of time or do you feel it genuinely added value to the company to engage in that exercise?

Sorry for the double post, accidentally hit reply twice on mobile client.

You can delete the comment within the first two hours, I think. Certainly can be edited.

Genuinely added value. I was firmly in the "NO" camp when I got to AMZN. Writing was something that people who didn't understand hustle did. :)

What I found, over time, was that the exercise of writing forces critical thinking. PowerPoint is corporate theater. Very easy to convince with your voice instead of facts, logic, data, and insight.

There were certainly times where I felt that the doc process was used unnecessarily as a blunt cudgel, but for the big decisions, and the planning process, it was incredibly helpful. As I have transitioned to working with companies as an investor, advisor, and board member, I have continued to use the templated FAQ docs for driving decisions. I am starting to get my teams to use them for board meetings.

IANAL but you should be careful about sharing private information about a large publicly traded company

Always appreciate the reminder. I'm answering questions at a high level and hoping to share the learning from my time at AMZN. Anything confidential will remain as such.

What is it like being in the stack ranking system? Is it as stressful as it is reported or have its drawbacks been exaggerated in the media?

If you come prepared, it's not as bad as you think. Being stack ranked +you're not in the room for that) can lead to stress though. I would say my experiences at MSFT under Gates and Ballmer for stack ranking felt far more capricious than at AMZN.

How did you feel about Kindle jailbreaks? Why not just let people get root if they wanted on the device they paid for?

Don't need to work at Kindle to know that publishers demand the drm, not a require by Amazon itself.

You can get Kindle books on an Android app on a rooted phone, or on a PC, or on a browser.

Nothing about DRM requires Amazon to jail the device.

I'm very interested in this 6-page memo, I also found during my PhD that writing can really help clarify thinking. You may think you have a very clear idea in your mind but then when you write this to paper, many gaps can suddenly appear. I've looked all over online for an example of a memo but I couldn't find anything beyond a basic narrative structure.

You mentioned below that you have templated FAQ docs, I'm assuming these templates are the approximate structure of the memo? Would you be willing to share? This would be very helpful for my startup when we are planning our strategy.

Sorry for the late reply. The template is more formatting than anything else. The core questions:

1) What is in this doc? What decision do we need to make? 2) Why is this important? 3) Who is the customer for this solution? 4) How have others tried to or solved this and how were their solutions insufficient? 5) How are we planning on solving this? 6) specifics 7) What options have we considered and thrown out? 8) What resources are required? 9) What are the one-way doors in this plan? (meaning: what decisions cannot be undone) 10) Timing?

Amazon is also pushing hard on gathering data. They've offered free photo storage for prime users for some time; earlier this week, they offered me a $15 gift card just to install and setup auto-upload of pictures from my cell phone.

I thought Amazon's empire rests on AWS which actually funds the company allowing them to make a sustained loss on their retail business despite attempts to automate with robotics and ML.

AWS is now pushing Alexa for Business, which I assume is a corporate wide AI product along similar lines as IBM's Watson.

It's basically the opposite? The retail business funds all the other projects. That's starting to shift, but only recently.

https://www.fool.com/investing/2019/04/07/how-amazon-costco-...

https://www.businessinsider.com/amazon-web-services-cloud-on...

AWS makes all the money, everything else is marginal.

> Amazon Web Services, which accounted for about 89% of Amazon's $1 billion total operating profit this past quarter. That's despite AWS accounting for just 10% of the company's overall revenue during the same time frame.

> But despite all of this, Amazon generates just a fraction of its net earnings from e-commerce. Instead, Amazon's bread and butter is its cloud computing segment, Amazon Web Services (AWS).

No, it's the opposite.

https://www.fool.com/investing/2019/04/07/how-amazon-costco-...

https://www.businessinsider.com/amazon-web-services-cloud-on...

AWS makes all the money, everything else is marginal.

> But despite all of this, Amazon generates just a fraction of its net earnings from e-commerce. Instead, Amazon's bread and butter is its cloud computing segment, Amazon Web Services (AWS).

Anyone has a link to the full article?

https://outline.com/34PN5W

"Screw ad supported services, if only they provided a way to pay for their content, then I would totally just do that"

-- HN

It adds up fast. I spend 100 euro on all my news subscriptions (and i still don't have the NYT or the WP). I can totally appreciate that others may not subscribe as much, or to the same set of services.

Spotify for news would be super good, though.

> Spotify for news would be super good, though.

Apple News+ [0]

[0]: https://support.apple.com/en-us/HT209513

Yeah, I appreciate the thought, but I have no intention of getting invested in the Apple ecosystem.

I guess this is making a virtue out of necessity. Amazon is clearly behind other tech companies in deep learning, so they are calling it "low-key." I used to work on the retail side, and their prediction process is pretty bland. They use an off-the-shelf neural networks, but they don't know how to do modeling, so forecasts are inconsistent. I am sure they will try to incrementally solve the problem, but proper modeling is foundational to predictions so I think it will be a long time before it is resolved. Like I said, it is about average in industry.

Reply on news.ycombinator.com