Besides the demands of agile and corporate culture, is there other reasons why data science folks are leaving (if true)? For instance lack of cooperation, or not using the insights/recommendations?
A full-time data engineer should probably have been hired.
I wonder if this leads to an analogous discussion as http://wiki.c2.com/?ArchitectsDontCode.
You told me last year that one data scientist is gonna solve all my problems. Now you're telling me I need to babysit him with five data engineers, because he is not going to get his hands dirty with data??
Who would sign up if they told you up front that your main contributions would be shuffling email, filling out TPS reports, attending pointless meetings, and being roadblocked every time you tried to automate, eliminate, or improve that drudgery? You can smash your head against the wall trying to make things better, but all that ends up doing is pissing you off and giving you a headache.
The biggest problem can be summarized like this: If you want to run your business differently, let the new people you hire do something different. If you want to do the same old thing and just act you are doing something differently, just teach the old people to talk like the new people.
I've got plenty of anecdotes from the year, but here's a simple one. The division had an existing "Research and Analytics" group that used linear regression in Excel to develop models. They would develop models on awful (read: highly-multicollinear) data with 100 rows and get "R-squared of .98." Large long-term business decisions were made on these models.
Over the course of the year I fought this constantly. Why do you need more/better data when the other team is okay with the current data? Why are you struggling to get your R-squared to >.90 (output of any type of model had to be R-squared) when the other team can do it so easily? Why do you need to speak to so many people to understand the data better? What I realized over the course of the year was that nobody actually cared about the models unless they matched the "gut" of senior management. So it was easy for the other team to p-hack until they got a slidedeck people would like.
I think there are plenty of problems that lend themselves to good Data Science. These are ones where you can properly identify what a prediction or better understanding of data (e.g. variable importance, coefficient confidence intervals) means in terms of resources (e.g. people, money, time). You can also identify what it means to not have that data insight or model.
There are also plenty of business areas where the link between resource investment and long term success is less clear (e.g. Marketing, Sales Incentives) and more gut and smart work than science. A lot of people are trying to clear up that grey with data science, but I go back to what I said before: you need to let people think and try different things over time to see the value of those experiments.
Comically enough, the Research and Analysis team picked up all the lingo we used and became a huge success: work the old way, talk in the new way. Management decided their Data Science adventure was a failed experiment and we just hadn't lived up to the hype.
I joined a non-tech company as their first data scientist, and am now tasked with building up the data science efforts of the company. Your comment is really ringing true, though I've found the company here to be generally great to work with in terms of their flexibility. Funny enough, it took me months of working with the IT department to get a VM with 8-cores and 64 GB of RAM, or the ability to spin up AWS instances (none of the analysts at the company had ever asked for that). I'm happy with it though, it's a fascinating challenge and I'll admit at times I feel a bit in over my head.
In my experience, many companies are wanting to hire a lot more data scientists (and paying them six-figure salaries to boot), when what they really are looking for are what has traditionally been called a "data analyst" position. I.E., the entire sum of the job is Excel reporting and basic statistics, but instead wrapped in some "Machine Learning" marketing language.
As opposed to more long-term focus on what business problems can be solved by carefully cleaning data, building models, testing predictive power, and then deploying these models into production. Which obviously takes a lot more time, a lot more effort focusing on a single problem, and doesn't have nearly as many immediate deliverables that higher-ups can open their Outlook inbox and see. Naturally they then wonder why they are paying such a premium for someone who doesn't bring as many deliverables to the table as their 60-70k analysts. And then the data scientist wonders why he's spending most of his time sending Excel attachments. Definitely more of a problem in non-tech companies.
Most businesses think small. I'm not even sure it's about risk/reward, either. I think people really don't have vision especially when it comes to technology and more specifically data.
The next generation is coming, though.
Having some expertise with BI applications (e.g. Tableau) has been a major strength for showing value.
100% Agree.
Some of the execs I reported into were obsessed with the idea of being a startup within a large organization, and part of this was the assumption that one of the things startups did was use data better than big companies. To them, there was some hidden value in their data that their Business Analysts did not have the skills to unlock. Data Scientists are perceived as the magic key to unlock this value, even when what the business is asking for doesn't make sense or isn't feasible in the short-term.
One of the small ways we delivered value was to write small ETL scripts that fed into Tableau dashboards. People loved the Dashboards because it cut down on so much Excel stuff that was done weekly/monthly/quarterly, etc. The company did have a large IT org with a BI team, but they were always busy and took months to do anything. So for us to do ETL+Tableau in a few days was seen as a miracle. Plus, Tableau has an Excel export :)
Of course, ETL+Tableau wasn't enough to retain the team.