The Data Science loop: the day-to-day work of a data scientist

The essence of what is a Data Scientist boils down to this awesome tweet by Josh Wills (@cloudera): https://twitter.com/josh_wills/status/198093512149958656

Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician.

It you want to make an impact as a data scientist, it helps to prototype your ideas as a testable product. Many product managers and engineers ignore their data scientists' work because fighting fires almost always takes precedence. The recent HBR article (posted here) showed how the data scientist at LinkedIn was able to make a huge impact by running his own experiments against live users. At my company, we always provide a non-optimized prototype that can be run directly with our existing product. If the idea is good, we can optimize the code later.

I would be interested to see more "low level" posts about how people approach their data analysis workflow. I do data analysis for a living and find that efficient workflow has just as much impact on your productivity and results as the choice in technologies that you use.

It would be great to hear more about the communication section.

For instance is it better to offer an ambitious summary first, followed by a deep exploration into the data and methods to understand how any conclusion may have been determined?

I am loathe to put something out there might overstate a point that will be run with without a sincere understanding, but I also fear that if the details are provided first without a thorough explanation of the nuances, that there is a danger they will be used carelessly.

I suppose my question is, have others have found it preferable to start with a headline then support that headline with caveats, or to create a detective story and walk people through arriving at a conclusion?

It's little like the conundrum of TED Talks (or a much of science news reporting). A very highly produced talk will establish an argument for an important adjustment of conventional wisdom in some field, only to create new problems because to people outside of the field it sounds like the final word.

This post completely neglects the part where data scientists derive statistical models from large sets of data in order to use them for classification, clustering or prediction purposes. Describing data science as advanced applied accounting is only true if you have to 1) analyze big datasets or 2) find hidden connections between data aka. data mining

Thanks for the post, I am starting a role as a data analyst soon and am curious to hear more about two aspects of your role:

1. How are the dynamics between engineering team, product managers and data scientists. Understand that LinkedIn's data team plays a huge role in creating and improving features (like People you may know), how is it like at Facebook, and in general other companies?

Engineers build things, managers make decisions, data scientists answer questions

2. What are the tools you use ( I see R mentioned in the thread), what books would you recommend? I mostly write in python, use D3 and gephi for visualization. Am also taking the Coursera course on Social Network Analysis (https://class.coursera.org/sna-2012-001/class/index) and reading the course book Easley & Kleinberg, Networks, Crowds and Markets. Thanks for sharing again.

intersections are nonsexy.

ask the hawkers who ply their wares at intersections. not only do they have to deal with traffic from the north, there is a constant barrage of traffic from the south, not to mention the incessant traffic from the east, and hey, how can we forget the speeding traffic from the west...

but the same intersections are also frequented by pedestrians from the north, and the west, the south and the east, so the hawkers' trade is lucrative.

data science is at the intersection of linear algebra, machine learning, statistics and distributed computing.

if you ask the hawkers sitting in a nicely furnished airconditioned shop at the mall, they will tell you that the hawkers at intersections aren't real hawkers, they are just fly-by-night hustlers selling a stalk of unkempt roses that will wither away, selling evening tabloid newspapers that'll be useless to read tomorrow, unhygenic ice-cream cones and candy, unhealthy street food, braids for the hair that'll snap if you tug at it, fake watches, imitation handbags, ... so if you want to be a real hawker selling real healthy food, you must open a real restaurant in a mall. you wanna sell genuine cartier watches, get a license and open a premium retail outlet in a mall. you want to hawk useful literature, open a barnes and noble bookstore in a mall...

so also the genuine statisticians will mock the data scientists as fake...oh these guys don't grok industrial strength SAS and S-PLUS, they fool around with unproven toys like R.

the genuine linear algebraists are too busy submitting academic papers to the MAA to worry about trivialities like data science.

the genuine distributed computing programmers know that data scientists operate with a very tiny subset of distributed computing - usually just hadoop or bigtable, and even that they dodge with syntactic sugar like pig, cascading and scalding. they are not even real programmers - they don't refactor their code, some of them just write ad-hoc scripts that they don't even check in to the repository, they don't do agile, hell they don't care about readability of code - they call their chebyshev decompositions "def cbd()" instead of "public static void chebyshevDecomposition( DenseDouble2DMatrix inputMatrix)", how can you trust these jokers...

the genuine ML guys work on world-changing technology like genome sequencing and autonomous vehicles, natural language processing and credit card fraud detection, not fluff like mining information out of tweets and facebook likes and linkedin profiles and foursquare check-ins.

but you see, the same intersections are also frequented by pedestrians from the north, and the west, the south and the east, so the data scientists can put food at the table and make rent :)

How do you become a self made data scientist? What are the approiate books, courses, majors to study?