Comment by jerzyt | Hacker News Reader

Comment by jerzyt | Hacker News Reader

Parent Full thread

A better question would be why AI works great for some business (e.g. Netflix, AirBnB, Uber, Waze, Amazon) yet fails miserably for other (JC Penney, Sears). In my view, the older companies are trying to strap on AI on top of a traditional dataset, which never collected any useful signals. The new companies designed their entire business concepts around data, and collected what's needed from the get-go. Sears may have a 100 years worth of useless data. AirBnB has about 13, but so much more informative. Amazon applies A/B testing all the time - would anyone at Sears even know what it is?

A secondary issue with business data is that vast majority of the features are categorical, for example: vendor id, client id, shipper id, etc. These usually get hot-one encoded, and you end up with hundreds of features where there's no meaningful distance metric. Random Forest and XGB are about the only that produce somewhat rational models, but in reality, they are good because they approximate reverse engineering of business process.

And lastly, the hype far outweighs the possibilities, at least until the business are ready to re-engineer the processes, if it's not too late.

AI works for online businesses where you have millions going to a single interface and thus any testing is on a random selection of the population.

You couldn't do that as well with different stores simply because malls have different demographics (meaning any conclusions could be noise) and the costs of shuffling where things are located is high so you can't just test 50 different setups and see what works.

Sears has been around for about a 100 years, and they have invented the catalog sales model - they were the Amazon of that era. I don't blame them for not doing things like A/B testing a 100 years ago, but 10 or 20? They were asleep at the wheel. That's when Walmart took off like a space rocket. Essentially the same offering to the same demographic. Yet Sears collapsed.

There were other factors at play in the downfall of Sears than just failure to take advantage of their positions at the time. Sears, in a sense, became a victim of its own success. In the heyday of parasite capitalism, it became more profitable for a small group of bad actors to make Sears fail.

Xerox is a more apt example of the 'missed opportunity' narrative.

The big tech companies all deal with this issue as well. One of the big problems when I started at Google in 2009 was that an experiment would show a mild negative effect on click-throughs when what was actually happening is that it was a mild positive effect for users but broke logging on IE6, hence resulting in a 0 CTR for that population. They solved this by building a system that automatically sliced results by population, alerted immediately if any one population was a serious outlier, and displayed sliced results on the experiment dashboard.

The big old-line brick & mortar chains just didn't think it worthwhile to build this sort of granularity into their systems, and are paying the price for it. I suspect that many executives who grew up in the 50s-70s think in terms of "Is this change good or bad?" vs. "Why is this change good or bad?" (Note that brick & mortar retailers who have embraced extensive data operations - notably Walmart, Target, and Safeway - are doing great. It's the Sears & JC Penneys of this world that are failing.)

> Amazon applies A/B testing all the time - would anyone at Sears even know what it is?

I wouldn't be so certain of others ignorance. Retail stores have long done studies and applied consultants to problems on layout, music, pricing, etc.

The problem with Sears is not that they have bad technical leadership. Their problem is they were purchased by a vulture who attempted to extract all of the wealth from the company, in a short period of time, for personal gain.

A number of senior Amazon folks went over to Sears 5-7 years ago. Sears definitely had people that know Amazon's techniques.

Your post makes some really salient points, and then misses the head of the nail in my experience. Sure, most of the information Sears historically collected is probably junk for supervised learning models. That isn't what makes this hard. High cardinality categorical features aren't what makes this hard. Re-engineering the business processes aren't what makes this hard.

What makes this hard, is that for all of these companies, machine learning models are essentially used in place of heuristics of varying degrees of complexity. The models are being used to incrementally improve heuristics that in some cases are tuned quite well. Couple that with the issue that a machine learning model is only one piece of an actual product improvement for these websites/companies (ie: now you have a prediction, what are you gonna do with it to effect the product?), and all of the sudden you have actual incremental improvements completely misaligned with moonshot expectations.

We've had plenty of rankers and recommenders for decades, the current wave of deep learning variants are improvements, but they're incremental from a high level. If your business wasn't successful/is failing using simple heuristics (sears, jcp, etc), a machine learning model isn't going to magically correct that.

> AI works great for some business (e.g. Netflix, AirBnB, Uber, Waze, Amazon)

Does it actually? Specifically, what has AI done for Uber?

https://eng.uber.com/machine-learning/

Oh, they use for marketing to investors, i understand.

Do you have any evidence that they are lying about all of the use cases they list on that web site?

Reply on news.ycombinator.com