Until recently I led a Data Science team (manager and IC) at a non-software Fortune 10. I, and everybody else on the team, left within a year of joining. The division where I was in hired me to grow both Data Science practice and the team. The company was enormous, so there were data science teams all across the company, all operating in silos.

The biggest problem can be summarized like this: If you want to run your business differently, let the new people you hire do something different. If you want to do the same old thing and just act you are doing something differently, just teach the old people to talk like the new people.

I've got plenty of anecdotes from the year, but here's a simple one. The division had an existing "Research and Analytics" group that used linear regression in Excel to develop models. They would develop models on awful (read: highly-multicollinear) data with 100 rows and get "R-squared of .98." Large long-term business decisions were made on these models.

Over the course of the year I fought this constantly. Why do you need more/better data when the other team is okay with the current data? Why are you struggling to get your R-squared to >.90 (output of any type of model had to be R-squared) when the other team can do it so easily? Why do you need to speak to so many people to understand the data better? What I realized over the course of the year was that nobody actually cared about the models unless they matched the "gut" of senior management. So it was easy for the other team to p-hack until they got a slidedeck people would like.

I think there are plenty of problems that lend themselves to good Data Science. These are ones where you can properly identify what a prediction or better understanding of data (e.g. variable importance, coefficient confidence intervals) means in terms of resources (e.g. people, money, time). You can also identify what it means to not have that data insight or model.

There are also plenty of business areas where the link between resource investment and long term success is less clear (e.g. Marketing, Sales Incentives) and more gut and smart work than science. A lot of people are trying to clear up that grey with data science, but I go back to what I said before: you need to let people think and try different things over time to see the value of those experiments.

Comically enough, the Research and Analysis team picked up all the lingo we used and became a huge success: work the old way, talk in the new way. Management decided their Data Science adventure was a failed experiment and we just hadn't lived up to the hype.