The Rise of the Data Scientist
flowingdata.com
flowingdata.com
What books would be on the required reading lists? What is the best set of tools to learn and know?
An alternative approach: how much of this could be automated and delivered as a web application/service? Provide easy ways for people to parse and upload their data, algorithms to run with good descriptions and recommendations for lay people, and well designed visualization tools for presenting the output of the data. This would be a very valuable service, but would take a broad range of knowledge and expertise to implement well (experts in the CS, Statistics, and Visualization fields as described in the article).
Computer Science (Data Mining, Machine Learning):
- Hard science book like PRML from Chris Bishop or Neural Networks from Haykin
- Fun implementations from Programming Collective Intelligence by Toby Segaran
- See practical implementations by these people: http://www.doloreslabs.com/
Statistics:
- Good foundation of probability theory and calculus
- Understand these slides: http://www.autonlab.org/tutorials/index.html
Visualization (something I'm working on but not good at yet):
- Check out http://processingjs.org/
- Dabble in Adobe Flash/Flex Framework
You will also find that a lot of the top data scientists/information visualisers actually come from an artistic background, e.g. http://www.visualcomplexity.com/vc/ and http://www.vimeo.com/5091290
The original article inspired me to go back to university so that I can become a data scientist. Can't wait.
I'm sincerely curious.
There are a few masters courses that could lead you to a career in data science, but it depends on your background. Most of the courses I found were specialisations, e.g. a masters in computer vision or HCI for those with a computer science background, or a masters in digital media or design and technology for those with an arts background. I had neither, so I thought I might have to do a conversion course into computer science or art and then specialise. However, when I read the article above, I realised I was more on the design side anyway (happier in Photoshop and Illustrator than a text editor) and that I needed to learn the computing and statistics side. This brought me to bioinformatics since I already have a background in biochemistry. The course will teach me programming, data mining and statistics. I also get to do three projects, which I am going to do in visualisation of life science data - which is why I got interested in this topic in the first place.
I think undergraduates have a greater opportunity to 'cobble' a course together tailored towards data science.
I found it useful to do a bit of research on anyone who does a visualisation that you like and see what their background is.
I'm coming at it as someone who was doing NLP a decade ago, and recently came back to it by taking courses part time here at Carnegie Mellon (where I am employed). So I have courses under my belt now covering the intersection of machine learning and NLP (which is pretty much all of NLP these days) and planning to take the Masters Machine Learning course this Fall.
The part I would need to add next, I guess, is the data visualization part.
For the large scale data part, there is this program:
I was already thinking about applying to this program, and the discussion in this thread is further convincing me that it's a good idea.
I think data science is a very broad subject area that covers the science of organising information (information science and vlis), right through to information visualisation.
Information science would have given me a good grounding in the skills I need for my start-up (which is involved in organising research information), but I thought it would be best to build on my expertise and interests (design, HCI, UX, psychology) and team up with someone who knows a lot more about the technical side of building information systems - more than I could ever learn from a one year masters.
It sounds like we are starting at opposite ends of the data/information science spectrum; it will be interesting to see if we meet in the middle after our masters.
Then of course for the actual data processing and visualization you'll need software such as Matlab, R, Processing, or Python (with Scipy + Matplotlib).
You'd be better off writing enterprise software in this field. Especially considering anyone who would pay for data munging and visualization tools/services probably doesn't want their data out on the internet.
Maybe not a perfectly "general" solution, but there are lots of domains where data is analyzed that would benefit from a powerful web-based analytic environment.
these datasets are usually huge
Not all interesting datasets are huge. Some large data sets can easily be shared among users -- Amazon already hosts a lot of common data sets in EC2 already, for example. Given the steadily-decreasing cost of bandwidth and the fact that you don't need interactive response (at least for the initial data load), I don't think it's that impractical for a pretty wide set of scenarios.
Might be a dual product strategy, the public facing webapp with limits on data size to test drive, and customer site installed enterprise version. Or maybe some solution that uses Amazon storage (in other words, outsource data storage and security issues to Amazon).
Of course, Google had the two tier search idea (they wanted to monetize by selling custom search servers to businesses) but found the real money was to be made on the consumer facing side. So you never know.
One of the things I am hoping to do with my start-up is to build simple tools to help wet-lab scientists access informatics solutions to analyse and visualise their data, and simple tools for informatics scientists to visualise their data so that a wet-lab scientist can understand it (more easily). There is a huge communication gap between (bio)wet-lab and (bio)informaticians, and I think 'data science' tools can help to bridge that gap.
http://www.meryl.net/2008/01/175-data-and-information-visual...