Hacker News Reader

Comment by SworDsy | Hacker News Reader

if i work for facebook and i want to figure out something about my users, isn't it safe to say N = All since the data im accessing is all user data from fb? it's easy to go wrong with big data, and although the article glossed over some fairly important things (assuming the people who work on these datasets are much dumber than they are in reality), they're right on about idea that the scope and scale of what big data promises may be too grandiose for it's capabilities

View on HN

Whilst, in the example you provide, it might be the case that "N = all", the cautionary tale offered in the article is that you always need to make sure you are asking the right question, and it is pretty easy to confuse yourself.

So you said "if i work for facebook and i want to figure out something about my users", and for whatever you were doing, looking at your existing user base might be the right thing to do. Perhaps, though, you actually want to know something about all your potential users, not just the users you happen to have right now. Whether or not your current user base offers a good model for your potential user base would then be a pretty important question, and one that almost certainly isn't answered by "big data".

I think that, as with most of statistics, the key point is "think about your problem", and that focusing on a set of solutions rather than the problems themselves can get in the way of that.

Even if you have the full population in question and thereby avoid sampling issues, you still have a lot of pitfalls. For example if you just start correlating every variable against every other one and picking out ones that hit some test of statistical significances as "findings", you run into a range of familiar problems generally grouped under the pejorative term "data dredging".

At first I thought so too. But it's actually easy to come up with cases where N != all. As a radical example, Facebook preserves the accounts of dead users.

Reply on news.ycombinator.com