How Big Data Became So Big

The buzz around big data, let's not forget, is partly fuelled by bullshit generated by vendors of big data solutions and analytic firms.

It probably does not really need to be repeated, but I guess it cannot hurt. Data being big rarely causes it to be much more interesting. There can be much more actionable information in a few hundred records than in billions, depending on context, proper experimental design, and so on.

Also, an often overlooked point is that precision of statistical estimators (e.g. width of confidence intervals) typically scales with the square root of sample size; yet the cost of processing an additional observation is typically constant (e.g. with distributed systems like hadoop). So in many cases there are decreasing returns to data size.

Good look at the history of the marketing term big data. I think this fits squarely in the realm of big data hype, or at least a review of why it is big data hype (something I've blogged about here http://www.klaviyo.com/blog/2012/07/16/the-curse-analytics-b...). The SAS discussion makes it especially clear that it's viewed as a trend.

But - posts like these frequently jump to the top of hacker news, and there's clearly a feeling that access to new analyses and more data is going to change the world. I think this is probably right. I'd love to hear more discussion of how big data has already started this process.

Examples I can think of (and I'd love to hear more): - weather analysis for farmers (weatherbill / the climate corporation) - marketing (the target pregnancy example - http://www.nytimes.com/2012/02/19/magazine/shopping-habits.h...)

More examples?

Painful reading an HN front page article like this. An entire article around use of two-word (marketing) nomenclature with no informative value in the article at all...

I'm already getting allergic to the term. Every couple of months there is a new term for the same thing to fuel a new hype cycle (machine learning, computational intelligence, data mining,...).

Not that it isn't an important thing, and certainly more and more data is becoming available. Now lets get back to work.