Data mining without prejudice | Hacker News Reader

Data mining without prejudice

web.mit.edu

56 pointsllambda16 comments

Original paper (no paywall): http://obs.rc.fas.harvard.edu/turnbaugh/Papers/Reshef_Scienc...

If you want to play around with the actual goods in addition to reading the paper there is a jar file and a R wrapper for same at http://www.exploredata.net/Mine/index.asp

If you are paywall-blocked you can get the paper at http://turnbaugh.openwetware.org/Publications.html

I wonder how much of the $15.00 it costs to rent this article for 24 hours the authors of this paper will receive.

Is that a serious question? In any case, the answer is ‘none’. The way academic publishing works, authors of papers do not get paid at all (though they may have to pay “page charges” to the publisher in some cases). That’s one of the reasons so many of us think the whole system is ripe for reform.

People often think that academic journals are providing the service of 'publishing' but they are in fact providing a reputation management and curating system. Not a great one, mind you. The fact that the end product is a journal is just an artifact.

Actually come to think of it, small record labels provide a similar gatekeeper, tastemaker kind of role.

So.. they ran mutual information on a correlation network with an window optimizer. Did I miss something?

It looks like a neat tool, but how does this rate `Science`? (as anything other than a section in a broad review of non-parametric data exploration techniques - of which there are many)

This boils down to a following old algo: 1. plot all possible scatterplots 2. order them due to an area of white space left 3. claim the top-whatever novel exciting relationships. This way it is anti-robust and produces too much false-postives to be seriously useful.

I think you can correct for the false-positives to some extent. I am more skeptical of the computational load of basically testing all sorts of relationships. They have an approximate algorithm for doing that.

Yeah... do it exact and no hit will survive; do it with Bonferroni and nothing will really change. And in fact it is only checking two-way relationships, which is already a minor fraction of "all sorts".

I love these techniques but wish they'd market themselves differently. This is hardly able to detect true relationships between variables---the false positive rate will likely be astronomical even with their MCMC p-value table on the website---but it is a more efficient way of exploring a high-dimensional data set than just comparing it blindly.

I think put in the camp with Hadley Wickham's Grand Tour methodologies, this is pretty interesting. Outside of that, it's just a well-marketed method jut waiting for failure.

Fascinating read, can't wait to look into it more closely. So far this approach seems somewhat similar to CHIRP [1] in that both concepts use bins or grids to find the most effective pairwise feature combinations.

[1] http://users.cis.fiu.edu/~lzhen001/activities/KDD2011Program...

I believe the full paper that this article is based on can be found here:

http://www.sciencemag.org/content/334/6062/1518.full

duplicate of an older thread: http://news.ycombinator.com/item?id=3364077

Tried this on a set of data that I have been looking at recently, and was not impressed. The software actually assumes all fields contain numerical data. Why not handle nominal data? Also doesn't handle temporal data. Maybe this is theoretically interesting but practically it is useless.

Is it possible to transform the temporal and nominal data into something with a numerical scale? For example, seconds since a certain date, or number ranking of the numerical data.

Or just not as generalizable as you'd wish?

Reply on news.ycombinator.com