Big Data Needs To Think Bigger | Hacker News Reader

Big Data Needs To Think Bigger

techcrunch.com

31 pointsjoshbert5 comments

I find these articles to be long on puff, short on content. The take home is: big data is really important, and some people think their startups will make buckets of money either storing/ analyzing it or selling software to process it. I don't really need 800 words (and a bunch of unwanted javascript and tracking cookies) to tell me that.

Here's what I would like: 12 technologies or techniques for handling big data, and why, with examples. Or a tutorial on a complex analysis that merged 3 really big datasets and came up with something useful (I dont need Hadoop to count words, thanks). Or a list of mathematical background every data scientist must have. But I am afraid the people who read these articles don't actually know "their asymptote from a hole in the graph", and that they don't have examples of big successes (maybe not yet, I grant...)

My daily job is merging building permits and tax assessor parcels to get small area population estimates. This seems like "big data" to me (though nothing like, say, all the twitter messages or google searches or FB friend connections); if someone could explain to me why -- and how, exactly -- I should move away from my SQL queries on and string matches, I would be really interested, but I all read off of this hype is BIG, BIG, BIG, AUTOMATICALLY DISCOVER EVERYTHING, MAKE MONEY, BIG, BIG, BIG.

Just my little rant...

Are your SQL queries problematically slow, or do you expect them to be in the foreseeable future? If so, can this be fixed by throwing beefier hardware at the problem?

Unless the answers to these are yes and no, respectively, then your data is small enough. Now, there may be usability reasons for going with different database technologies, but from the lack of panic in your post, it sounds like you're doing fine.

(Incidentally, if you've got data problems that aren't being satisfied by SQL queries, then you could probably get a really interesting discussion going if you wrote about it.)

The video demos on Palantir's analysis blogs are pretty mind blowing.

http://www.palantir.com/government/analysis-blog

Why do we even care about "Big data"? Why do we need to count every single datapoint? We don't do that anywhere else - The census, medical studies, insurance, engineering, etc.

We use stochastic and Monte-carlo processes to get an answer.

Here's a nice concrete article on things you can do with one particular dataset:

http://norvig.com/ngrams/

Reply on news.ycombinator.com