Comment by stiff | Hacker News Reader

Comment by stiff | Hacker News Reader

I am not sure if I disagree with the central thesis of the article, but the argumentation seems rather weak in many places, for example he says:

Let's take the example of the number of people who walk into a store and buy a certain product. By studying 10 customers you can get an estimate of the probability of the purchase and predict sales; by studying 100 customers you get a better estimate. By analyzing 1000 customers you have a very good estimate. You could study a million customers, but the results are unlikely to be vastly different from the results of analyzing 1000 customers and probably not worth the expense.

This makes sense only for very low-dimensional data sets, but most real-life problems are very high-dimensional and in high-dimensional spaces a million customers (or any other number of samples you might reasonably expect to gather) might still very well be only a small sample. So, there are situations where because of various constraints you will never be able to get to this level where more data stops yielding better results. You can watch a nice presentation from Peter Norvig about the unreasonable effectiveness of data:

http://www.youtube.com/watch?v=yvDCzhbjYWs

Or read about the well-known curse of dimensionality:

http://en.wikipedia.org/wiki/Curse_of_dimensionality

That specific point stuck out as incorrect to me as well. And given that it's one of the only technical points in the article, doesn't really inspire confidence in the author. That said, however, I do tend to agree with the overall hypothesis that the value add of "big data" is very marginal for most established industries, for exactly the reason he mentions: they've all been doing this for 20 years now. It's just become cheaper and easier recently. There are certain domains where modern data techniques can have a transformative effect, he mentions one (law). I don't think the data analytics solutions and companies that are being talked about are capable of generating these kind of transformative innovations though; so, I think the author is absolutely correct to say that the big data hype is overblown. There _will_ be big opportunities for big data, but they will come from disruptive startups finding new ways of doing old jobs. Not from crunching a few more GB of your data for a few less bucks.

Good point. Also in most situations data collection goes far beyond 1,000 customers at little to no additional cost. This doesn't necessarily break the author's point though since if there was tremendous value to this data, programs like Google Analytics wouldn't be free. The fact is that most of this data is worthless. You get most of your value from one or two data points that you chose to quantify a qualitative goal.

My perspective comes from my first market research job where I was hired to "make sense" of Nielsen's data for a TV series. It was a cop-out on the company's part - a trained high schooler could have done what I did, organizing thousands of numbers to recognize the 4-5 key takeaways to present back to them. I think the intimidation of large sets of data often tricks people into thinking they're more valuable then they are, and as data collection intensifies, so does this fallacy.

Reply on news.ycombinator.com