What Wal-Mart Knows About Customers' Habits (2004) | Hacker News Reader

What Wal-Mart Knows About Customers' Habits (2004)

nytimes.com

46 pointsheinrichf22 comments

> Three years ago, Wal-Mart summarily announced that it would no longer share its sales data with outside companies, like Information Resources Inc. and ACNielsen, which had paid Wal-Mart for the information and then sold it to other retailers.

Anyone have more information on these types of data feeds? What PII is usually in them? They're an opaque, behind-the-scenes practice that I'd like to know more about. Wal-Mart may not have given them anything for a decade, but I'm sure tons of other retailers do.

> What PII is usually in them?

It really depends. There are a lot of dynamics at play, and depending on the source of the transaction data it may or may not have identity data tied to it, or be in a format that would easily be identifiable when combined with other data sources. Some sources would consider identity data itself as a competitive advantage, and only sell anonymous or aggregate data to others. Other sources would have no value out of withholding the identity data, and would capitalize on the opportunity to include it in their feed for a premium.

Feel free to email me if you want to learn more (email in profile). I'm not an expert on consumer data, but used to source business-related data feeds in a past life and got a glimpse of the consumer data side of things.

Zero PII. On occasion there are loyalty programs and IDs but there is no link back to the consumer. Companies like IRI and Nielsen recruit their own panelists and build models of the consumer that they then project into census style data.

Yeah, i don't know about that. There is a data feed I know of that contains the last 4 credit card digits...

Think of it in terms of checkout data: a transaction ID and itemlist with UPCs. Customer data is a whole 'nother enchilada.

I did some assignments for what I believe was ACNielsen when I was in college. I'd get paid about €10 an hour to stand outside an Aldi store asking every customer that came out for their receipt. (Aldi also doesn't share its sales figures.) I'd also have to note every customer that didn't give their receipt. This gave ACNielsen information on the sales for that particular store. They, of course, did this at multiple locations and sold the gathered data to interested parties.

> By its own count, Wal-Mart has 460 terabytes of data stored on Teradata mainframes, made by NCR, at its Bentonville headquarters. To put that in perspective, the Internet has less than half as much data, according to experts.

~15 years later, 460TB are about 10.000 euros in cheap over-the-counter hard disks.

or 70k € in SSDs.

I find this statistic dubious at best, given that by 2004 the internet had hundreds of millions of users. Even if we're generous and assume they mean public data on the internet, that'd be less than 1MB of data per person.

How many people were passive consumers then though? Also, hough much lighter weight was the content? This is before smartphone videos were being uploaded after all, 2004 was still an era where you might have a page on your personal site for your dog and everything was static html + css.

I guess its subjective by what they mean by "the Internet has less than half as much data". I imagine what existed on file sharing sites was more (if that counts). As of 2007, I had 1 TB of specific types of content, which would have been a tiny tiny fraction of what was on sites like morpheus/kazaa at the time in 2004.

As a very rough ballpark, hard drive cost was around $1/GB, putting a single copy of everything around a half million dollars. Honestly quite a bit less then I expected for back then. Of course there are tons of other costs, but still not unreasonable for a Walmart size company.

http://www.mkomo.com/cost-per-gigabyte

That 460TB of data has grown exponentially with the data brought in from stores, and from our growing e-com side over in Sunnyvale.

If I had to place a number, it's well over the ExB barrier.

Hey guys! WM Associate here. Let me know if you have questions about our data storage practices (or anything else).

I'm wanting to shed some light on our practices, as we aren't as sketch as most believe.

Throw-away account for obvious reasosns.

Purchase data sharing with 3rd parties and facial recognition,how rampant are they? And do you know anything about how they track cell phone's wifi mac as a customer goes around the store?

Data Sharing Agreements: None that I know of. That bit is a little out of my range.

Facial Recognition: Very, Very Little. Only in one store at the moment, and it's designed for Asset Protection and tracking a thief around the store.

As for tracking MAC addresses, we track Enterprise devices that associates use, but not customers.

Why is PULSE such a steaming pile?

Because our internal infrastructure is a steaming pile of s*. Probably. I don't use it, so I can't particularly comment.

Does 1 person's weekly buying habits affect a store?

I feel like everything I buy has gotten more variety since I started shopping there.

Also, how did you find out about HN?

Not really, many of the stores are $1M+, so a $100 purchase of groceries could just be considered an outlier. The bigger changes you'll find is when word of mouth spreads about a specific product. Ex. If Tide came out with a new version of the pods, sales will be slow (assuming no advertising) for the first few weeks, but it acts more like an exponential curve if the product catches on.

Good! The Dark pattern for the store layout is working :P

Friend referred me.

Maybe it was an attempt to ward off Amazon, also Target.

Walmart has spent a shit ton of money on their analytics systems including open source and Teradata. They have agreements with various cell companies for geo-fenced info. So they have location analytics and your "market basket" data to slice and dice. Yet the results are mediocre and Amazon is kicking their ass with strategy. They have no loyalty program (Sam's has membership) to provide compelling packaged deals to shoppers, they have their Every Day Low Price pretense but check Neighborhood Market prices that vary wildly from super-center prices especially for grocery. In reality, WMT is a quasi-monopoly that is dying under its own obesity - it needs to split up. eCommerce at $16B is a joke considering it doesnt make any profit and has take billions in overpriced investment. The management, especially in tech, barely see projects through and has lately removed several long-time employees for "fresh blood". Add turnover in SV and other places, it is chaos.

This'll be fun to dissect.

> Walmart has spent a shit ton of money on their analytics systems including open source and Teradata.

This is true! More specifically, we're using Teradata and Hadoop.

> They have agreements with various cell companies for geo-fenced info.

First that I've hear of it. If we actually did have this, it would be a major invasion of privacy. There was a pilot to be able to track expensive inventory (Like TVs) around the store to prevent theft, but it was shutdown due to privacy concerns.

> Yet the results are mediocre and Amazon is kicking their ass with strategy.

Go look at our year-over-year for e-commerce and you'll see that we're actually doing very well.

> They have no loyalty program (Sam's has membership) to provide compelling packaged deals to shoppers, they have their Every Day Low Price pretense but check Neighborhood Market prices that vary wildly from super-center prices especially for grocery.

This is where it gets interesting. Most of the variation you'll see is taxes from zone to zone. As well as logistical challenges for certain locations. Given actual places, I could get a breakdown of costs, but alas.

> In reality, WMT is a quasi-monopoly that is dying under its own obesity - it needs to split up. eCommerce at $16B is a joke considering it doesnt make any profit and has take billions in overpriced investment.

Do you watch out our financial calls?

> The management, especially in tech, barely see projects through and has lately removed several long-time employees for "fresh blood". Add turnover in SV and other places, it is chaos.

Management is terrible, I'll agree to that. But HR isn't ageist, and we have thousands of associates that are > 25 year tenure with Walmart and hundreds of those over 35 year tenure.

Turnover in SV is because we meet the bare minimum to compete in the market, not because of management, otherwise you'd see the similar numbers to our offices in Reston, Virginia.

Thanks!

Reply on news.ycombinator.com