The new reality of business intelligence and big data discovery

applieddatalabs.com

13 points·chuckmans3··7 comments

> This type of reporting is fantastic if you’re technical and understand it, but for the average businessperson it’s reminiscent of that calc class they barely passed.

I like how the response to that is that the reporting needs to get better and not that average businesspeople need to smarten up and pay attention in math. I don't disagree that reporting can and should get better, but this is a shared responsibility.

Big data has always reminded me of the warning Jaron Lanier gives about the Turing test. Paraphrasing, it is "the only way for a computer to win the turing test is if you make yourself dumb enough for it to pass".

Big data, and the expected revelations, feels like that to me...

Does anyone have any tips for finding the "needles in haystacks" in the middle of a pile of big data? The fact that you don't know which questions to ask seems to be the biggest roadblock in effective deployments.

O'Reilly's book _Data Analysis_ is very good.

What qualifies as "big data" though? Is it a terabyte? A petabyte? Or is it just a buzzword?

I work in BI and data analysis and this is my personal definition: traditionally, companies have mined their own internal data. Cost to sales, balance sheets, customer surveys, top sellers... Etc.

Big data are defined as those data external to the organization. My definition of big data is to also look without as opposed to only looking within.

It's a buzzword but it's also meaningful. If you've ever tried to look at data that doesn't fit on reasonably sized server you'll understand.

Reply on news.ycombinator.com