Hacker News Reader

Comment by MattGaiser | Hacker News Reader

AI works for online businesses where you have millions going to a single interface and thus any testing is on a random selection of the population.

You couldn't do that as well with different stores simply because malls have different demographics (meaning any conclusions could be noise) and the costs of shuffling where things are located is high so you can't just test 50 different setups and see what works.

View on HN

Sears has been around for about a 100 years, and they have invented the catalog sales model - they were the Amazon of that era. I don't blame them for not doing things like A/B testing a 100 years ago, but 10 or 20? They were asleep at the wheel. That's when Walmart took off like a space rocket. Essentially the same offering to the same demographic. Yet Sears collapsed.

There were other factors at play in the downfall of Sears than just failure to take advantage of their positions at the time. Sears, in a sense, became a victim of its own success. In the heyday of parasite capitalism, it became more profitable for a small group of bad actors to make Sears fail.

Xerox is a more apt example of the 'missed opportunity' narrative.

The big tech companies all deal with this issue as well. One of the big problems when I started at Google in 2009 was that an experiment would show a mild negative effect on click-throughs when what was actually happening is that it was a mild positive effect for users but broke logging on IE6, hence resulting in a 0 CTR for that population. They solved this by building a system that automatically sliced results by population, alerted immediately if any one population was a serious outlier, and displayed sliced results on the experiment dashboard.

The big old-line brick & mortar chains just didn't think it worthwhile to build this sort of granularity into their systems, and are paying the price for it. I suspect that many executives who grew up in the 50s-70s think in terms of "Is this change good or bad?" vs. "Why is this change good or bad?" (Note that brick & mortar retailers who have embraced extensive data operations - notably Walmart, Target, and Safeway - are doing great. It's the Sears & JC Penneys of this world that are failing.)

Reply on news.ycombinator.com