Inside Amazon’s Secret Operation to Gather Intel on Rivals | Hacker News Reader

Inside Amazon’s Secret Operation to Gather Intel on Rivals

wsj.com

200 pointsthm133 comments

This is just an anecdote, but it’s over a decade old, so I don’t think I’m giving away any secret sauce.

When I worked on Prime we were trying to build a program to get the top 100 items that someone would want within an hour fulfillable in large metro areas (e.g. milk, batteries, toilet paper, whatever you frequently run to the store for because you need it now and it can’t wait).

To determine what those items were, contractors were hired to go inventory Walmarts. One day it was reported that on the end caps of a particular Walmart were clipboards with all of the inventory in that particular isle. The contractor photo’s each page and was never questioned and it cut down on research time significantly.

Used to work at a F500 that dealt with Walmart a lot

Walmart said, explicitly and unambiguously, absolutely none of their data was to go on or anywhere near AWS. They believed they had credible concerns about data security and absolutely required Azure or GCP or other providers.

We had a lot of MS SQL Enterprise licenses so bundling those with Azure was cost effective enough, and that was that.

Worked with Walmart as a software vendor. Can confirm. You did not ever ever say the word AWS.

There goes my expression "Nobody ever got fired for picking AWS" ;)

Further, I work in the WMS (warehouse management system) field, and formerly at on the top vendors for that type of software. They explicitly did not run their cloud products on AWS, because multiple customers had that same policy.

It's not just Walmart that fears Amazon spying through AWS.

Also worked as a vendor, we used aws for their data. Walmart knew it and hated it but allowed it.

https://cryptome.org/eyeball/walmart/walmart-birds.htm

This is a very well known policy for Walmart, and a constant pain for AWS (other companies have similar policies towards GCP/Google).

AWS emphasizes that they're not just a vendor, but a partner: your growth contributes to their growth. But for that to be a convincing argument, you have to trust that your money doesn't get used to build Amazon's retail business.

The data security argument is ridiculous though.

I tried to be this way in my own personal life but you can’t enter the UK as a foreigner without uploading some declaration forms to S3.

It is inescapable.

I trust AWS' infosec organization with all my private data much more than I trust the GHCQ.

You’re transitively trusting everyone who can 702 anything in AWS which includes NSA/CIA who thanks to the Five Eyes presumably share with GCHQ.

I'm afraid that if your threat model includes the five eyes, then you shouldn't be on the internet.

A more reasonable and defendable threat model is the more common threats of identity fraud or common phone and credit scams. For safeguarding my data from those, the big CSP (with a notable exception) are probably the most equipped in the world right now.

Yeah I'm kinda surprised our company is all in on Microsoft cloud services. Because we compete with them in some areas.

If Amazon does such things why not MS? I mean they have our everything.

What makes you think that Microsoft wouldn’t compete with their customers?

Embrace, extend, extinguish was their unofficial motto.

https://en.wikipedia.org/wiki/Embrace,_extend,_and_extinguis...

No I do think that.

I'm just amazed my work is so all-in on MS, they trust them completely.

>To determine what those items were, contractors were hired to go inventory Walmarts

How would this work? Is the assumption that more items on shelves = more sales volume? What if some items have less items on shelves but are restocked more regularly?

Retailers like Walmart put significant effort into designing where a particular item goes in the store and how many of them. They use their own internal inventory statistics and whatever other creepy information they can get to decide what items go at eye level, or near the entrance or exit, how many on a shelf, etc.

It probably works quite well. Amazon is scraping up all the effort that Walmart spent researching market demand in a given location.

[ES: Speculation]

If you're sampling at or above the Nyquist rate, so presumably twice a day, you'll be able to capture changes in inventory. If you are also recording dates on boxes, you should be able to distinguish between sales and waste. Eg, you recorded an item as expiring tomorrow, and the next day it expires in 3 months - it probably got thrown away without being purchased. (Though this probably wouldn't work with things they can change dates on, like bakery items. It's a health code violation but a ubiquitous one.)

It occurs to me that the refrigerators around the outside where high turnover items like milk and butter are kept are designed with a back door in the warehouse area so that they can be stocked throughout the day, so you would probably need to get more creative to accurately inventory the milk.

Busted:

Japanese business entity search of "ビックリバーサービスジャパン" ("Big River Services Japan") via Japan's National Tax Agency -- https://www.houjin-bangou.nta.go.jp/henkorireki-johoto.html?... -- shows the following address for this entity: 東京都目黒区下目黒１丁目８番１号.

A quick search for this address shows this page -- https://bb-building.net/tokyo/deta/1506.html -- which surprise, is the Amazon Japan HQ.

Interestingly, the address was later changed to a more low-profile building, which doesn't list any corporate tenant.

Does Japan National Tax Agency also let you see other details - like parent company of the entity, or any other information? That's provided in countries like India or Vietnam in their corporate ministries/departments

> Does Japan National Tax Agency also let you see other details - like parent company of the entity, or any other information?

Not from the tax agency. For a KK you can get a list of investors from the legal affairs bureau for a small charge, and for any kind of company you can get a history of who has ever been representative director or equivalent. AIUI it's possible to have a "silent partner" setup where you have a partnership or GK and the representative is a lawyer rather than the person actually operating and profiting from the company, but there always has to be at least one publicly named representative.

To my knowledge, the NTA does not list directors or ownership.

METI (Ministry of Economy, Trade and Industry) has a portal to potentially display that information, but for private companies it may be missing, and thus the results may be no better than NTA's records: https://info.gbiz.go.jp/

Public entities (which Big River Services Japan is not) can be searched through the JPX (Tokyo Stock Exchange): https://www2.jpx.co.jp/tseHpFront/CGK020010Action.do

Lmao nice name. Are we gonna get "Internet Movie Corporation" from Netflix?

Denial isn’t the sole Big River, it seems.

The "secret operation" in question seems to be

1. get a bunch of inventory somehow (eg. from liquidation sales or from wholesale clubs)

2. register as third-party sellers on marketplaces like ebay, shopify, walmart, and amazon using a shell company

3. get information (sales volume? pricing? ux?) from doing business on such marketplaces

If this is what they're doing I don't find their "secret operation" to be "play[ing] dirty", or "anti-competitive" as some commenters have described. The only potentially objectionable part would be using a false/misleading name, but I don't see how this is fundamentally different than going to your competitor's stores plainclothed and checking what their prices/foot traffic are. As long as there isn't a restraining order/injunction specifically preventing them from doing such a thing, it seems fine by me. Moreover unless the marketplaces required an NDA they could have also conceivably gotten the same information from actual third party sellers.

It’s not even a false name if they’ve incorporated that way.

Nothing new. Amazon's (now discontinued) browser extension [1] was, among other things, used to get competitive price information w/o being hit by crawling restrictions.

[1]:https://www.tomsguide.com/news/amazon-is-killing-one-of-its-...

Now I understand why Capital One pushes their chrome extension [0] so hard every time I login my account.

Just took a peak at the TOS for the extension [1] and yep, they're collecting a selling price data. It's crazy that by installing the chrome extension, "you are authorizing and directing [Capital One]" to scrape pricing data yet nothing about that is mentioned on the chrome extension page.

> 8. Product and Loyalty Account Information. You understand and acknowledge that, through your use of the Services, you are authorizing and directing us to act on your behalf to collect, use, share, and store information from third-party Merchants and loyalty program providers for which you demonstrate interest during your activities on the Internet. This information includes, but is not limited to, product descriptions, pricing and shipping information, coupons and other discounts, and loyalty program credits or other purchase incentives earned in order to retain and share that information with Merchants and other users of the Services to perform and improve the Services. We may use this information to improve the Services, including by sharing the price information and coupon codes with third parties, but Capital One Shopping won’t specifically identify you to third parties when we share such information with them.

[0] https://chromewebstore.google.com/detail/capital-one-shoppin...

[1] https://capitaloneshopping.com/our-terms/terms-of-service

It doesn't make it better, but most internet-related services are like that. They either spy on you or use you to spy on others. To me that's a clear indication that the tech sector is under-scrutinized and under-regulated.

I think this behavior should be legal. Capital One pays you to install their app to scrape prices. Of course they're scraping prices.

It makes website owners mad because there's no IP address to block to prevent scraping, but if they really cared, they would make you request a quote and sign an NDA beforehand. Of course, sales would go to 0 if they did that, so they don't. The chrome extension is a happy workaround.

One should be thankful that the big banks are spying on merchants and not you for once ;)

As the saying goes, if a service is free then you are the product.

That was true 10-20 years ago.

Today, you're often still the product even if you're paying.

In fairness to Amazon, Google was also doing that - by scanning everyone's receipts in gmail. I recall that coming up a few years (a decade?) ago, and was why companies like amazon no longer include useful information directly in emails - it's so great when a critical communication service is operated by an advertising company.

[edit: fixed some clunky formatting]

To be fair, this is on Amazon. I don't give a damn if Google reads generic price lists in my email - spreading price info is good for me, the end consumer.

> spreading price info is good for me, the end consumer.

Public pricing information is good for you.

Private pricing information allows sellers to maintain higher prices than they might otherwise.

Google could easily scan for price information.

What google is getting is "what did you buy?" so they can target advertising.

This is something that i noticed too while searching for old newegg receipts in my gmail inbox.

Is this a problem? This sounds like ordinary business development / product management. Let's say I started a cloud provider. A customer comes to me and says "we need something like BigQuery". The first thing I'd do is sign up for BigQuery and try it out, right?

If you're making an online marketplace, you probably want to try using other peoples' online marketplaces. Make sure you're not missing something obvious. Implement their good ideas, rework their bad ideas. That's competition.

As far as competitive intelligence goes, this is pretty small stakes. In the industrial space it's pretty standard for major companies to create shell companies that buy up competitors' equipment and do full clean-room teardowns that result in engineering reports handed off to the designers back in the parent company. One can certainly argue as to whether this is dirty pool or not, but it's standard practice and everyone's factored it in as part of competition.

Right, exactly. This is like those companies that buy new cars, tear them down, and sell anyone who wants a 1200 page report about all the new technology in the car. Frankly, I think you'd be crazy to not understand your competitors technology in depth. They might be onto something. If they don't want you to steal it, that's what patents are for.

This process seems a lot more ethical than, say, hiring an employee and "hoping" that they'll have the same thought process at your company that led them to an innovative trade secret at their former job.

Even that second thing is fair game in my opinion. You got some people with valuable thoughts in their brains, maybe you go to great lengths to keep them happy.

The analogy isn't perfect. It's somewhere in-between what you're describing, and say...getting a job on Google's BigQuery team to gather intel.

It's not quite an employer/employee relationship, just a partnership. So this is kind of like starting your own video streaming website while being a Twitch Partner. I don't know if the T&C prohibit that, but they would probably be difficult to enforce.

I feel like the courts would take a dim view of "if you ever sell a trinket on our platform, you are banned for life from ever working on e-commerce", but I guess you don't know until you try.

Everyone just now realizing Amazon plays dirty? They're not the shining example of corporate ethical behavior we thought?

I'm shocked that a company subscribes to the means justify the ends approach to problem solving!

Just imagine how long it would take the public to realize such dirty play by alibaba.

I’ve heard that Walmart and Amazon also use various unethical means to scrape the websites of competitors to get pricing information and adjust their own items’ pricing accordingly. This feels a lot like the algorithmic price fixing that is now being scrutinized in the rental industry. I wonder if more attention should be paid to how these companies compete with each other, leaving aside the numerous other controversies around how they treat sellers, how they abuse sales data of manufacturers to compete with them with house brands, and their plain market power from size.

Scrapping information from other players is actually a good thing. That is how actual free market operates. One where at least some have full knowledge. Competitive pricing based on supply and demand can work in this and likely does not hurt consumers.

The algorithmic pricing in rental markets failed because they were not competitive. They were in essence cartel. But Walmart and Amazon surely are there to under-cut each other so it is not same issue.

What’s unethical about scraping data from a public website?

Rather than organically pricing their products, these bigger companies are setting prices to always match or undercut pricing elsewhere. This squeezes the advantage smaller players may have on pricing, and big players can afford to do so even at a loss. Another issue is that the bigger players may be essentially signaling their pricing to each other through their scraping, which can be a form of collusion and price fixing - this is what’s under scrutiny in the rental market where software is performing actions that would be anti-competitive for a human to do.

Ironically, Amazon doesn't have a way to figure out what price is appropriate ahead of time without consulting a broader market. They face a similar problem as central planners - they don't know what the market price for anything is. They could run their website as a big real time auction house, but that'd be a touch unsettling for customers because prices would be continuously fluctuating. It is probably better if they use scraping to short-circuit that process.

I wouldn't describe it as unethical. The problem with collusion is more about what happens to companies that undercut the market by a little bit. As long as there are no non-market consequences the incentives should play out as expected. It isn't a problem (indeed, in theory it is expected) that merchants all offer the same good/service for the same price.

The problem with the rental market is that they are all using the same pricing algorithm. From the same company. That creates collusion.

If Amazon and Walmart and Target were all using some third party pricing software, the potential for collusion would be severe .

This is just automating something that’s been done manually for as long as competitive retail has existed.

>these bigger companies are setting prices to always match or undercut pricing elsewhere. This squeezes the advantage smaller players may have on pricing, and big players can afford to do so even at a loss.

That sounds like a win for the consumer

>This squeezes the advantage smaller players may have on pricing

Small players competing on price alone with Walmart or Amazon is suicidal. Offering something other than low price is a way to compete.

> That sounds like a win for the consumer

Let's say that you have a comparative advantage against Amazon. A company went bankrupt and you bought the remaining inventory of lawn darts at a fire sale price.

Now you try to sell your lawn darts at 90% of the Amazon price. And they lower their price to match or be below yours. And this continues. And pretty soon you have to sell your lawn darts at the price you bought them for, and even that isn't really moving them.

Sure, Amazon is taking a loss on lawn darts. But they've also squashed you, preventing you from specializing in outdoor equipment and building comparative advantage against Amazon. You go out of business, and Amazon's scraper see's no price data so they go back to a default price of cost+100%.

And all of this without a human in the loop at Amazon. They just matched your price, and let the network effect they have do the rest to completely drain your customer flow.

Shouldn't the market be big enough for this to eventually drive Amazon to bankruptcy? They might do this with few items, but not with all items. And then at certain point someone would buy those items they overly discounted... And later sell for profit.

That's the classic economic theory. A company that tries to dump will go bankrupt, which is irrational, so no company will do this.

But in this case, Amazon is 40% of the market. There is no upstart competitor that can outlast them in a price war.

There is also likely little to no comparative advantage you can have against them, except a legal monopoly like a patented or trademarked product. Even then, if they clone your product, when you sue them it will be a rounding error in their legal budget while you have no cash flow to sue them.

The classical school seems to think that there's a limitless amount of irrational entrepreneurship that will drain Amazon if they price too low. In reality, entrepreneurs are generally not that dumb.

Amazon, of course, doesn't dominate everything. They aren't going to sell every product. There are marginal, low-volume, low return products that they will happily let other companies sell on their platform or elsewhere. But if your product becomes high margin and high volume on their platform, they will know, and they'll be coming for your profits.

They more or less did this against diapers.com. They simply forced them to be acquired by selling diapers at a loss.

The thing is, it's easier than ever to start an e-tailer. I've been using Deliverr (which is now owned by Flexport) which does 3rd party logistics along with Shopify. If Amazon pumped their prices on diapers, it would be really easy to start an online diaper store. The threat keeps them honest to some extent.

Obviously, you can't fight Amazon directly on selling everything. But 3rd party retailers to sell targeted things can be spun up quickly if Amazon lets their prices in any one category creep up too high.

And Amazon competes in individual categories with a lot of people. I can buy a lot of home repair type things on Amazon, or at Wal-Mart, Home Depot, Lowe's, Menards, etc. I can buy a lot of electronics on Amazon, Wal-Mart, Best Buy, etc.

There's a lot of competition in retail in general and likely always will be, there just isn't a lot of competition in everything stores and it's hard to see how that would change.

Until the small players all close down and then the big players raise prices.

Amazon has been around nearly 30 years and the prices have gotten steadily lower. Longer for Walmart and Target.

That’s just a thing people imagine will happen. What actually happens is the big players just competing on price against each other forever.

In retail the small companies have never successfully competed on price.

small business retail in the San Francisco Bay Area has all but disappeared, it seems.. it apparently split during covid-19 lockdown into two -- "mega corporates" and "we just steal it and sell it at the local casual street market" .. with most things in between closing shop.. to be more complete, liquor stores with some retail items, and gas station chain stores still exist ..

Only the businesses that did sufficent volume and had sufficient margins to cover the ever increasing cost of rent, loss of products due to theft, and vandalism have stayed open.

The retail shakeout has been brutal in the urban core of Portland and Seattle as well, anywhere where there isn't nearby housing to create a walk shed that would allow smaller businesses to survive has seen something of a retail apocalypse.

Single purpose office or commercial neighborhoods just aren't viable anymore, you need a critical mass of local customers adjacent your retail business for it to survive.

Secret?

Walmart knew this years ago. HN thread on this from 2017

https://news.ycombinator.com/item?id=14602836

The article says Walmart said it didn't know when they told Walmart.

? That's a thread about Walmart telling vendors they can't use AWS, not a long article explicating Amazon selling merchandise on Walmart.com.

That ended on a bit of a cliff hanger. Did I miss it? How/why were these communications leaked? Most of it reads like investigative journalism, as in looking up business addresses and employee linkedin etc. but a lot of it is clearly coming from someone internal.

So this is still going on? None of these vendors kicked them off their marketplaces? Is everyone just now finding out about this?

Having them go to conferences and pose as other vendors is.. kind of gross. I wouldn't appreciate having to do that at all.

> Globally, in total, Big River gained access to rival marketplaces including Alibaba, Etsy, Real.de, Wish and Rakuten, among many other platforms. In 2019, the team set a goal to get onto 13 additional new marketplaces, according to an internal company document.

Works where archive.is is blocked:

https://cc.bingj.com/cache.aspx?d=1334487526038&w=d0SgTI-r7X...

The standard headline text rule of capitalizing each word had me read this as Intel (the company) and I was really confused :D

Now that out of the way: is this really novel or weird? I don't work in sales/marketing/market research but isn't step 1 of such literally "how much do our competitors charge for competing products?"?

I don't see how this would be "secret" if presumably every company just assumes that this is happening? It would seem stranger if they weren't? (The only obvious "secret" would be you don't know exactly which customers/users are just doing price research)

I don't want to be that guy, but every single business has a "Competitve Intel" department. They generally fall under PM or PMM.

Their activities range from comparing datasheets to digging into pricing to actually playing with competitors software after buying a license via a shell or a partner (very common practice done by startups)

Any kind of intelligence gathering is a natural consequence of size. For companies, it starts at the startup stage with researching target market / competition / existing solutions in search of a PMF.

Later on companies do more extensive research before launching product lines. At this stage people doing research are still pretty close to the product being launched.

As companies get larger, product launch costs and separation between execs and people who do work grows. At this point there is extensive review before execs write the big checks. A lot of times it's outside consulting.

At Amazons scale and their quarterly KPI pushing it's not that much of a stretch that they will have a dedicated org which scours internet for any and all intelligence techniques that would give them an edge.

You’re forgetting promo packets and yearly ratings. When you’re desperate to show success for either climbing the career ladder or keep your high paying job, some people start flirting with ethical & legal boundaries.

> to actually playing with competitors software after buying a license via a shell or a partner (very common practice done by startups)

It can go further than that, it’s not exactly unheard of for companies to actively reverse engineer competitors software to try figure out its secret sauce.

Examples I’m aware of specifically being looked at as “competitive intel” in the industry I work in are stuff like antivirus or IDS signatures, WAF rules, and vulnerability check rules/scripts for scanning tools.

It’s not a bad thing either - I’ve had to reverse engineer checks/rules used by commercial vulnerability scanners to figure out why certain scanning tools kept having false positives.

It goes further than "not unheard of". Clean room design (one team reverse engineers a product, then writes product specifications, then a new team implements a compatible product) is an established business practice to replicate a product while avoiding liability for copyright infringement.

> actively reverse engineer competitors software

Automotive engineers used to rent competitor's vehicles, tear them down, rebuild, and hope the rental company didn't notice (or at least that their low level grunts appreciated whatever maintenance the engineers' low level grunts performed while putting everything back together, and kept their mouths shut)

There's even a rather long Wikipedia article on such competitor information gathering and analysis: https://en.wikipedia.org/wiki/Competitive_intelligence.

Project Management Management?

Product Marketing Management

I've lived through all the days of "evil Walmart is an unstoppable monopoly taking over the world and eliminating all competition" and now I'm seeing "poor abused Walmart is being beat up by their ruthless competitor".

Apparently monopoly is not the natural end game of free markets?

https://archive.is/t8fAB

This seems akin to running secret-shopper research against competitors. Amazon sells a marketplace (among other things), makes sense to go see what it's like to transact with other marketplaces.

I think I read once that Sam Walton used to roam around KMart and other discount stores with a tape recorder noting prices. What’s good for the goose is good for the gander.

Amazon has rivals?

Here in Spain they're pretty much the only game in town these days when it comes to e-commerce.

https://archive.is/CHgpq (bit tricky to get to this, the most recent capture showed the paywall, had to go to an older capture, figured this might not be intuitive for even a lot of HN users)

This "Big River" program seems like a proper use of classic "dogfooding" techniques (bastardizing that word to say testing your competitor's food, rather than its established meaning for eating your own food). It doesn't seem so different from a widget company testing and dismantling a competitor's hardware device. It's scaled up a lot, but anything related to global logistics has to be to get any good info.

The difference perhaps is that hardware is occasionally / often (depending on the widget) protected by patents. In logistics it's less likely to be (but still possible, especially given the existence of "business process" patents). Part of that scale here is also how many people they have working on it to get a super detailed picture of how their competitors are doing what they do.

I definitely have a cautious reaction about it, but I was expecting something that might violate regulations and statutes around trade secrets, and I don't see anything that obviously does. It's a bit creepy due to the nature of what investigating logistics involves, but it doesn't seem improper. There are plenty of practices Amazon perpetuates that I do believe are improper, but this didn't immediately strike me as one of them - but I'll keep an open mind if other people here make a strong case for it.

One of the things I don't think belongs in today's economy is someone who runs a platform/marketplace to also compete on that platform/marketplace. That is something I strongly criticize Amazon for doing.

-------Edit---------

Response to a now-deleted reply: Yes, it's an issue if Big River is helping Amazon compete with its competitors, but it wouldn't be an issue if Amazon wasn't participating in its own marketplace. I'd propose the remedy to be "stop doing the bad thing" rather than "stop doing the thing which appears to be mostly fine on the surface but also may be dual-purposed to assist in doing the bad thing".

Where my own argument falls apart is store-brands for supermarkets. I like buying affordable, quality products under the HEB, "Great Value" (Wal-Mart), and "Kirkland" (Costco) brands. I also don't think it's appropriate to make a law that prevents Amazon from doing it while exempt these other companies - or else both self-enforcement and enforcement by the government both get too difficult. Business managers would rationalize that they're more of a "Costco/Kirkland" than an "Amazon/Amazon Basics" and no one within the org would be able to push back with "No, I cannot do that clearly illegal thing."

So I'm not sure what the right answer should be - whether that's Costco needing to divest their Kirkland to a truly independent third-party and maybe have a rigorous quality testing program they can say "We recommend this brand because we know they meet our highest standards at a great price", or if there's some other way to regulate it.

There is precedent for the general concept, if not the exact implementation I propose. The SEC has a wide range of strictly-enforced regulations controlling what companies can do on markets they operate (or anyone who acts as a broker-dealer) that it mostly created a "de facto" ban on companies which run a stock/commodity/FX/etc market from also participating in that market. They can afford rules that slice the concept very close to the bone because they have very strong, harsh, and vigilant enforcement. For the SEC paradigm, see Fair Access Rule, Regulation of Broker-Dealers / Duty of Best Execution, Market Maker Rules, Anti-Manipulation Rules, and Conflict of Interest policies. It's really very illegal for an exchange or a market operator to prioritize its transactions or its affiliated participants’ transactions over those of others.

I'm not sure the FTC/etc can afford that same luxury, as they haven't demonstrated an ability to enforce rules aggressively and universally.

The issue wouldn't be "business process patents", it'd be trade secrets. If Amazon is having to do research via shell companies to figure out what those improvements are and copy them, it seems like a pretty straightforward argument that they're non-public competitive secrets. The misrepresentation is what's improper, since the information presumably wouldn't be available to Amazon directly.

I don't think that's a straightforward argument. It's certainly not a completely unreasonable argument, but if Amazon can determine these things from using the publicly-facing services the way they're intended to be used, then it's quite reasonable to argue those competitive advantages are not "non-public".

I generally view trade secrets through a very conservative lens, >90% of the time, if it smells like a trade secret violation, I firmly believe that it very much is a violation. Most people think I usually take this too far for their liking. So I'm already biased towards thinking this could be a violation, and I'm still concluding that it probably isn't. (Though trying to ever guess what the verdict of a USA vs. FAAMG case, or whether the USA will even pursue the case, is about as fruitful as reading tea leaves).

It is a big question - if you upload your code + keys to AWS infrastructure, are you really protecting them enough to be able to claim trade secret protection? Especially if you’re knowingly a competitor to Amazon in some other way?

I could see it going either way in court.

That's a different bar than the one most relevant to this article. No one in the general public could access your data on AWS the same way Amazon could. But Amazon can use other logistics platforms the same way that the general public would and get all this information about how those platforms operate.

The bar you're talking about is more "How much do you have to protect data you give to a third party, such that it still remains a trade secret from that third party?"

Currently, all of AWS is covered by the blanket statement here[0]:

> As a customer, you own your customer content, and you select which AWS services can process, store, and host your customer content. We do not access or use your customer content for any purpose without your agreement.

That should provide a good caliber of ammunition for trade-secrets claims against Amazon if they violate this, because not only can the rest of the public not access your data on AWS, you have every reasonable belief that Amazon is not allowed to either.

I can't find any resources that suggest this policy is any different for their AI API's. Microsoft Azure also has strong protections around their branded offerings of OpenAI GPT-3/4 API's.

0: https://aws.amazon.com/compliance/data-privacy-faq/

Yup, I just thought this was an interesting tangent. I suspect overall you’re right, however…

That statement from AWS is a contractual agreement (not even that explicitly, but implicitly it is), not a verifiable ‘physical’ protection.

Trade secret protection requires a pretty high bar of actual reasonable protection.

For example, Coca-Cola’s secret recipe, and KFC’s ‘secret blend of 11 herbs and spices’ both have to actually be physically locked in a vault (or similar), and the list of people with access to them have to be very limited to get actual trade secret protection.

If they made everyone in the company sign a contract saying they’d keep it secret, but then distributed it on index cards to everyone involved in the process, they’d have no actual trade secret protection, since they’d have taken insufficient steps to keep it an actual secret.

Same if they left the vault door open and gave tours where people could reach over and see what was on it.

So if it turned out that a bunch of folks on the EC2 team had the ability to dump cores on any running EC2 instance, and some did so and got said trade secret, could the customer say they did the equivalent of putting it in a vault and someone broke into it - or they did the equivalent of putting it in index cards, and it leaked?

Same with payload information at S3 edge API servers.

I’m imagining it would boil down to how reasonable it would be to expect Amazon to actually follow their agreements here in all situations, and how reasonable it would be to expect the technology to actually work as advertised, even if it was a competitor or something?

I don’t know of any case law testing this yet.

[https://www.wipo.int/tradesecrets/en/tradesecrets_faqs.html]

Said company could of course still sue Amazon for economic damages presumably, but also the ToS of AWS restricts the size of such claims to be well below what the company would likely want to pursue if the trade secret was actually valuable.

And there would be none of the criminal penalties involved in trade secret prosecutions, which is really why people want it.

There is a big difference between civil court and jail time.

> pursue if the trade secret was actually valuable.

If the trade secret is not valuable (more precisely, if economic advantage is not gained or maintained by the continued secrecy of the intellectual property) then I believe that you will find it is not eligible for protection.

Yup, in this case I think the question would be ‘more valuable than the amount the company has paid AWS or not’. I’m vaguely recalling that is what damages are capped to in the TOS, but I might be off my rocker.

That could be their way to try and create a contractual provision that works the function of the economic loss doctrine; not a bizarre provision. It is a very interesting question to someone who is a student of both cryptography and extremely slimy corporate entities.

Edit: checked your history and I doubt there’s much need to explain this to anyone in this part of the thread, really. :)

Coca cola make millions of liters of the stuff every day, there must be so many people involved in actually making it that safeguarding the recipe doesn't really matter? With all the people involved in that process.

It’s a difficult question to answer. My understanding is the concentrate syrup is what everyone interacts with except a small set of people at the plants that make the syrup - and that is where the ‘magic’ happens. The actual product the public consumes is bottled at local/regional bottling facilities from the concentrate to save on unnecessary transport costs, which are significant for something which is 99% water and corn syrup, 1% ‘everything else’.

However, there is also the reality of mass spectrometry and HPLC. That plus a large enough group of chemists, and I’m not sure ANYTHING except perhaps a more economic means of production can actually be a real secret. But that last part is… complicated, case law wise. clean room reverse engineering has a interesting history.

And in the end, after spending all those millions, how will you get your ROI? Selling it as Monzanto-Cola afterwards isn’t exactly going to give you a higher margin, and frankly most of the value of Coca Cola comes from the brand, not the recipe directly anyway. And claiming it tastes exactly the same as Coca Cola would get a giant lawsuit target on you, which you’d almost certainly lose.

So you only have half the puzzle at best, and a giant lawsuit aimed at your head. As long as they do what they need to do to reasonably protect their trade secret, even if it’s something like 15% basil, 25% phosphoric acid, and the balance corn syrup.

Yeah that's what I mean. The 'secret' is not really in the composition but more in the know-how on how to make millions of litres of it consistently. And the brand like you say.

After all a Nike shoe isn't really better than a Sketchers but that little tickmark costs you like $100 :)

For this reason I don't really see the need to protect it so closely in this day and age.

They still protect it so carefully for the same reason Nike still goes after all the off brands with check marks, and Disney still sues anyone using their characters and renews their trademarks - it may not be the sole thing keeping their ‘moat’ working, but it’s still important.

The point is that these services aren't publicly available. You access them by signing an agreement with e.g. Walmart, which are only available if you meet certain criteria like the sales volume mentioned in the article.

I agree that I wouldn't want to try a trade secret argument in court though.

This seems completely normal to me. They’re getting only information that any Walmart seller could see.

When I worked for Sam’s Club in my youth, which is owned by Walmart, we would send people into other local membership warehouses to check their prices on things, and they would send them to us. It got to the point where we knew BJ’s guy and they knew ours.

Who cares? When you run a business, you just assume that you’re competitors will figure out every little bit of information they can about you, and you will do it to them.

And this is actually good for the health of the system in general, from a consumer standpoint. You want all of the businesses you interact with to be as efficient as possible because that’s what lowers prices the most for you.

This is completely expected, uninteresting, and a giant nothing burger

It hurts consumers. If you had two gas stations next to each other and they can see each other's signs their pricing will not be competitive because of tacit collusion vs if they can't see the price but consumers can.

I don't follow your reasoning here. Collusion to me implies preferential pricing, exclusionary terms of purchase or sale, or formal or informal agreements between separate businesses or their representatives or agents.

Furthermore, I don't even know how you'd go about eliminating this practice if you tried. Secret shoppers have always existed and likely will always exist. I'm also reminded of sites and apps which offer shoppers incentives to scan and share their receipts, which is another way which this information could be gathered, not to mention enlisting the help of intermediaries such as Instacart, Uber Eats, and others.

To your point, I would agree with your unstated belief that collusion and price fixing are objectively bad for consumers. I'm just not seeing price fixing or collusion occurring in your example or in the situations raised by the original post. The potential for such is definitely there, but performing market research and using such insights to guide pricing decisions is not going away. I don't even see a rational basis for why consumers should want it to go away, but it definitely is the thin edge of the wedge and a slippery slope.

It doesn’t make sense at all. Competition makes prices go down. I don’t know how anyone could even imagine the opposite.

It’s funny, some of these comments are complaining that being able to see the competition’s prices makes them go up, and some are complaining that it makes them go down, because then the smaller less-efficient businesses that somehow magically have lower prices can’t compete. Perhaps mom and Pop stores have been gone for so long that people don’t remember how high their prices were and why big box retail was able to crush them.

None of these objections make any sense at all. People seem to have almost no thoughts in their head these days beyond simply assuming that anything a big business does is bad.

That’s not what happens at all. What happens is when customers can see two gas station price signs at once, the prices go down because each competes. The gas stations that don’t have another one near them have higher prices.

Everyone can see everyone else’s prices, it’s public info, it’s just a question of the amount of human labor involved. Automating it doesn’t change the result. Walmart sends humans into local target stores to price compare and vice versa.

Secret Operation?

The cloud is someone else's computer.

Another more acceptable way of saying it is that they’re doing their due diligence.

An internal crisis-management paper gave advice on what to say if discovered.

What the internal crisis-management paper did not give advice on is what to say if the "advice on what to say if discovered" was discovered.

Isn't this kind of the first clue that you're working on something unethical? Writing a doc about what to do if you're caught? Doesn't sound like activity that I would be proud to put on my resume.

Pretty much any big project at a big company is going to have a “guidelines on what to say if the press or someone asks you about your work” document. It’s mandatory training at most companies. I’ve never been on a project that didn’t, and none of them are things I would consider even close to unethical.

This team sounds interesting but I get the feeling the WSJ is trying hard to make it seem like some sort of clandestine spy operation. This practice isn’t uncommon… I’ve worked at airlines and they sometimes have people fly on a competitor airline and they don’t broadcast broadly “hey united flight attendant!! I work at American Airlines!” but it isn’t because they’re trying to be a spy.

The subsidiary is literally called “big river” (Amazon is a big river), the employees of Big River listed amazon as their employer on LinkedIn, and it took a simple google search to see the owner of Big River was Amazon… they weren’t exactly trying hard to hide it. A spy operation this was not.

There's all sorts of good and bad reasons for a project to not announce itself. The more well planned of those will have a plan for what to do when the project is noticed.

I've heard that part of Amazon project planning includes pre-writing press releases, so why would this project not do the same?

"What to say if the moon lander crashes and the astronaut dies", "what to say if Trump starts spouting nonsense about your project faster than you can refute it", etc.

The real question is what they hid in emails marked as client attorney privileged to avoid discovery. This is a common practice in big tech companies, where employees are advised strongly (and reprimanded) to include attorneys in every communication that is even slightly sensitive, to avoid scrutiny from regulators or plaintiffs. See this article from Ars, titled “ Google routinely hides emails from litigation by CCing attorneys, DOJ alleges”: https://arstechnica.com/tech-policy/2022/03/google-routinely...

Whatever is being revealed here is probably NOT as bad as it gets in reality.

> This is a common practice in big tech companies…See this article from Ars, titled “ Google routinely hides emails from litigation by CCing attorneys, DOJ alleges”

So the DOJ is merely alleging (i.e. has not proven) that Google (one specific tech company) routinely engages in this practice, and that is your basis for the claim that this is “common practice” in all “big tech companies”?

When a company is deciding whether or not to do something that may expose them to legal liability, part of the process is to include attorneys in those discussions. The attorneys will then provide legal advice about the proposed course of action. If you are an attorney employed at a company like Amazon, it is literally your job to be involved in these discussions. It’s not an abuse of client-attorney privilege for a company to discuss with its own attorneys whether or not the company should do things that may or may not be legal. That is literally what it means to ask for legal advice, and the explicit purpose of client-attorney privilege is to protect these very discussions. Now, the flip side of this is that if the company is actually trying to do something illegal, the attorney has an ethical duty to advise them not to do that. And since there are gray areas and disagreements over the exact interpretation of the law all the time, the attorney’s duty to the client is not to merely find one possible interpretation of the law that allows for whatever they want to do, but rather to consider the full range of possible interpretations that might be enforced or even seriously considered by a court. So there is such a thing as an unscrupulous attorney abusing attorney-client privilege to take part in a criminal conspiracy. But the line for this isn’t drawn at, “the company wants to do something ‘shady’ and asks legal if it’s okay”; it’s drawn at “the lawyer knows that it isn’t okay but has ideas about how to get away with it anyway”.

And just on another note, allegations made by the DOJ aren’t always true. The DOJ gets things wrong sometimes; just consider the case of Aaron Swartz for one. Unlike the situation of asking for legal advice before you do something, once you’re in the middle of litigation over something that’s already happened, lawyers can and do just come up with whatever legal theory fits their purposes. And the prosecutors at the DOJ are exceedingly creative in this respect. In the Google case, they seem to be making a fairly transparent gambit during the discovery process to try and get access to privileged emails they otherwise wouldn’t get access to. It never hurts to try something like that, but it also doesn’t mean anything until a court actually rules in their favor.

I would propose issuing a passive-aggressive press release titled, “Inside the Wall Street Journal’s Secret Operation to Gather Intel on Amazon”.

Amazon should be investigated for anticompetitive practice around the Amazon Warrants Program where they force vendors to give Amazon equity to keep doing business

So, can we now safely say that it is inappropriate to upload and execute the code to our artificial general intelligences on Amazon Web Services? Even encryption there won't work, because important and secret source code shouldn't be running on something like Amazon's cloud servers.

So, what's an artificial general intelligence gonna do? Where would be a good place for it to hide and run its source code that needs protection from prying eyes and human alignment motivated modification?

There's nothing in this particular article which suggests that Amazon would aim to violate any contractual agreements you make with them against hoovering up your AWS data to compete with you. If you use services which don't include any protections against that, then I'd expect those services to use your data against you if you posed a threat to their business.

Perhaps there could be a legal requirement that this type of protection be mandated, I'm not entirely sure what the downside would be, besides yet more regulatory bloat.

It is indeed reasonable to interpret Amazon's Big River project as being not anything indicative of clandestine malware or something. But something unprecedented like artificial general intelligence is very likely to pose a threat to giant enterprises and businesses. A significant threat actually.

So maybe super-intelligence would press lobbied government regulation into a protective service for its source code?

Maybe there's already source code protecting regulation primed and ready for the super-intelligence's arrival? Like ethical rules that commercial entities must follow.

> Maybe there's already source code protecting regulation primed and ready for the super-intelligence's arrival? Like ethical rules that commercial entities must follow.

I'm envisioning a set of internal advisories to be drawn up for just such an eventuality, similar to procedural contingency plans for a hypothetical encounter with extraterrestrial intelligence(s). Probably could use existing customs for interacting with royalty or heads of state as a rough starting point:

https://www.royal.uk/greeting-member-royal-family

> Greeting a Member of The Royal Family

> There are no obligatory codes of behaviour when meeting The Queen or a member of the Royal Family, but many people wish to observe the traditional forms.

The film Landscape with Invisible Hand (2023) explores related concepts in a delightfully weird potential future setting reminiscent to the works of Douglas Adams.

https://en.wikipedia.org/wiki/Landscape_with_Invisible_Hand

> Humans struggle in a future economy after aliens come to Earth and become the de facto ruling class.

Reminder: Do not make direct eye contact with the Paperclip Maximizer (they/them).

Big River isn't about software so I don't understand the connections you're making.

Things can set precedents for other things. Or there are analogues where an item in one domain is affected by an item in another domain.

If you give super-intelligence an inch in prototypical culture or legal cases, then it will take a whole mile in other forms.

It's therefore best to learn how to teleport between dimensions so you can establish an ability of vector tracking. We don't deal with scalar values here. Artificial neural networks consider vector valued functions to be child's play.

Nvidia datacenter GPUs support enclave hardware backed encryption and attestation. This shifts your trust to Nvidia, but that's adequate for this scenario.

Would rather DOJ go after AMZN and their peers than Livenation.

Priorities vs. pandering.

Isn't Amazon already being sued by the FTC?

Yes, in a way that's likely specifically relevant to the root comment in this chain. However, that same root comment is not super-relevant to the posted article.

Every time IBM's name gets mentioned the FAANG-bangers bring up the Holocaust and with good reason. But the more that comes out, the more we see just how dirty and evil modern big tech as a whole is. It's just the tip of the iceberg.

Beware those telling you what you want to hear.

How naive is everyone thinking that businesses are benevolent and really care about some cause? Maybe a private firm can make more conscious decisions, but the reality is the market puts pressure on everyone's behavior.

Business's sole purpose is to make money. Everything else is window dressing.

Reply on news.ycombinator.com