Hacker News Reader

Comment by josh2600 | Hacker News Reader

I don't know.

When I think of big data, the first thing that pops into my head is Insurance Actuarial tables, and that's not interesting to me. Are statistics suddenly the hottest and most interesting thing in the world because we can run experiments over larger datasets? Maybe, but I think that most engineers capable of doing the kinds of analysis these firms want would be better suited to harder problems.

Don't get me wrong, data analysis is important, I just wonder if the IEEE has a duty to encourage organizations like this or if they should be trying to influence kids back towards the "hard" engineering practices.

To be honest, as long as people are doing something that makes them happy, I'm not one to judge, but I do think there's something to be said for attacking things that are harder than statistics.

View on HN

What makes computer science a "hard" engineering and statistics not a "hard" subject? I'd hesitate to call statistics an engineering discipline, granted, but I object to differentiating statistics from CS on the basis of the word "hard".

I hope you're not trivializing statistics simply because of the elementary approach most schools tend to take for the first three or four statistics classes offered at the university level. I would argue that statistics can be just as difficult as computer science, and just like CS has intractable problems, so too does statistics.

If your idea of big data is acturial tables, you're not thinking big enough. Actuarial tables are largely still small enough to be handled in orthodox ways using orthodox software (SAS has a deathgrip in managed health--what I do for a living--and it's a 50 year old piece of software). In fact, most of the times I hear people say their data has a very high volume, it really doesn't, and the traditional methods of data analysis can handle it. Once you step into Facebook/Amazon/Google sized data, things change. And that's only looking at "big data" as large volume--the most common definitions of the phrase involve several other variables (see: the four V's).

Just because what's commonly done or visible is elementary doesn't mean the entire field is elementary. It would be like assuming CS isn't a hard engineering if all you saw were web designers/developers.

It is pretty boring. Big data doesn't necessarily mean big insights. So lots of times it can end up being a wild goose chase with management not realizing they are funding science experiments.

For any one thinking of it as a career, I highly recommend Nate Silver's - The Signal and the Noise: Why So Many Predictions Fail - But Some Don't

The impression I got from the "cs/se/stats" remark was it was the fusion of all three disciplines. A statistician can't sit down and pull data from a MySQL table, and certainly can't write a library to gather data on users on the site. As someone that's basically in this exact confluence point, I can tell you that there is very much interplay between the statistics and the software engineering and computer science disciplines.

Hmm. What kind of statistician these days can't pull data from a MySQL table? Even the almost-retired people I know have to interact with data sources in some SQL product or another.

HN is not "average". False consensus effect: http://en.m.wikipedia.org/wiki/False-consensus_effect

I hardly think I'm special for posting on HN. Have you read job adverts with "statistician" in the title? In order to do your day-job as a statistician you need to work with data products and tools which require more than MS Office type computer skills.

Data Science != Statistics.

Reply on news.ycombinator.com