Data-Visualization Firm’s New Software Autonomously Finds Abstract Connections
wired.com
wired.com
In other words, if I'm trying to find out why a plane crashed, and the first thing this system tells me is gravity, then it isn't a practical advancement.
I don't understand what is their cost in letting people try it before using the tool; unless of course it runs on their machines. Put it somewhere and let me try it with some simple data (e.g. iris in R).
Replace "questions" with "hypotheses" and "thought" to "cared".
I'm all for having a tool that can highlight a ton of interesting things in my data... I'd make a business case to buy that any day. But finding all the connections is so noisy compared to finding the right connections. That's the cynic in me.
The optimist in me is looking forward to when these kinds of techniques are just another R package I can load and turn loose on my data. :)
It can train models for you automatically, so you could loop over every model available and return the top ten predictions. Of course it wouldn't get around the need for feature engineering, but its theoretically possible.
Note - do not attempt this unless you have lots and lots of machines, as it will take a very, very long time.
It’s a bold statement, however by using algebraic topology Ayasdi has managed to totally remove the human element that goes into data mining — and, as such, all the human bias that goes with it.
---
Don't companies with massive datasets already do this in some way? Google's system for piecing together what people "really want" seems such a large, open-ended inquiry that it seems some automated insight discovery must be done.
Also, wouldn't a machine that generates automated insights require another layer of meta-analysis to be able to sift through which insights are actually usable? Using the example of sports statistics, you could generate an infinite number of trivial relations between players and plays and teams...it seems that at some point, a human with real domain knowledge has to go in and program a filter system, which seems to be about the same amount of work as the inquiry-generation that this software automates.
Finally, why all the emphasis on visualization? Visualization helps to illustrate to humans the possibilities of investigation...for a machine that can supposedly ask (and answer) all the correct questions, isn't visualization merely eye candy?
It'd be great to see a concrete example of this software in action. Perhaps feed it the NFL play-by-play data that was posted on HN a few weeks ago and see if it can generate usable strategy.
Gunnar Carlsson, the researcher mentioned, seems to work mainly on an approach related to manifold learning (roughly, finding lower-dimensional structure in high-dimensional data) that's based on algebraic topology. He co-ran a workshop on that last year at NIPS, one of the main machine-learning conferences: https://sites.google.com/site/nips2012topology/ . He's written some highly cited papers on the subject, though it'd be a bit of a stretch to claim he invented the area, since there have been workshops at least back to 2007, that one organized by a set of French researchers: http://topolearnnips2007.insa-rouen.fr/description.html
I would guess the part about removing the human element from data mining is putting an optimistic PR spin on the basic idea of automatically extracting lower-dimensional structure, which, if it works, should allow for less feature engineering. The emphasis on visualization makes sense in that light, if they're planning to sell it as a no-expertise-necessary system: feed it data and get interpretable-by-non-experts viz as output, with any complexity that would normally require "data scientists" being handled automatically.
http://www.nag.com/Welcome_iec.asp
Mik