Data Science for Startups: Business Intelligence

> the next step is being able to answer all sorts of questions about product health within your organization. A novice data scientist might think that this type of work is outside the role of a data scientist, but identifying key metrics for product health is one of the core facets of the role.

This strikes me as disingenuous word play. Whether or not the identification of key metrics is important, it's fair to ask whether it ought to be part of the job duties for a data scientist, and your answer to that question probably reveals more about your personal inclinations than about what is optimal for a given business.

Personally, my experience has been that most of the metrics and ad hoc reports that product- or business-minded managers ask for are junk wastes of time which, at best, create new surface area for fueling political debates about what's important, allowing different managers and cliques in the company to duke it out over whose priorities should be favored. Dressing them up as sexy data-driven "insights" is just one more tool for politics.

When the words "data scientist" come to mean "ad hoc business report jockey" then it seems like you might as well dissolve the question, because in that company, they've already chosen what they want out of data science-- arbitrary and whimsical political fodder.

In other companies, data scientists are actual statisticians. It can be really hard for business people to let go of the keys to the car, but a well-trained statistician knows that you have to devise you experiments or your proposed models based on whatever inference goals you have, which would be business goals in many cases. By taking a generic statistical perspective, you will home in on the metrics and model diagnostics that carry actual meaning for decision making, and many times you'll just pragmatically ignore all the business junk (not always, because some of it rests on useful principles, but a lot of it really is junk).

Anyway, it's such a loaded topic. It all depends on the private mental model someone has in their brain when they hear "data science."

Some people start thinking about rigorous statistical models, trade-offs between a frequentist test or a Bayesian setup, whether or not a machine learning model is appropriate, what kind of diagnostic would make sense for a special regression model... these people need to be given freedom to make choices that are statistically rigorous, and trusted that they understand the inference goals in terms of business priorities for the company (probably even better than the business people do).

Other people hear "data science" and they think of a sharp-featured 20-something wearing a sweater vest parachuting in the business meeting with a sexy slide deck on their Macbook, ready to spin whatever story is needed. These people ... should not be in charge of anything.