There is not, I suspect, any other solution but that we must train a whole lot more statisticians. This means we will need to give more credit, and authority, and probably pay, to people who choose to pursue this field of study.
There is not, I suspect, any other solution but that we must train a whole lot more statisticians. This means we will need to give more credit, and authority, and probably pay, to people who choose to pursue this field of study.
This comment is not meant to disparage anyone who considers themselves a data scientist. However, as someone who has advanced degrees in both statistics and computer science, employers and recruiters outside of R&D roles have shown very little interest in my statistical background aside from my machine learning experience. My experimental design, data handling (not just data cleaning, but data collection) skills, and theoretical understanding are rarely discussed. Statisticians are compensated much less than programmers--maybe deservedly so--but to that extent that I'm compensated the same as a "data scientist" who only studied computer science and didn't study any statistics, I feel like many employers, even those who benefit heavily from them, don't properly compensate statisticians.
At my alma mater, the poor wages have really hurt the statistics program as more students have decided to enroll in the "data science" (typically housed within business or computer science departments) programs. I think this is a really unfortunate trend because while those programs teach you how to implement gradient descent, do basic data wrangling in Python and R, make data visualizations, etc. they don't teach experimental design or the statistical theory that drives applied statistics. Perhaps as the supply of well-trained statisticians decreases and demand increases there will be upward wage pressure, but I think it's more likely that unqualified and inexperienced "data scientists" will continue to be shoehorned into these empty roles instead.
What do you think practicing data scientists should learn to be more effective? Experimental design? Something else?
I don't believe you need a PhD to call yourself a scientist, but I do think one common trait most scientists share is curiosity. To that end, I would encourage practicing data scientists who might not have a formal statistics education to not shy away from statistical theory. A solid theoretical understanding is what guides you when the questions and answers aren't clear--and I think the fundamental shortcoming of many data science programs is that they prepare their students for extremely simplified (and therefore unrealistic) questions with easily obtainable answers relative to what will be encountered in the real world.
I apologize for giving you an answer that isn't as coherent as I would like to it be. I tried not to be too verbose, but I think I failed at that anyway. I have a lot of feelings on this topic that I haven't fully articulated to myself yet. I could answer your first question if you're still interested, but as an addendum to my original comment, even though my statistics degree has gained me nothing in terms of career advancement or an increase in salaries or opportunities, its been truly invaluable to me as a programmer and a public speaker and advocate for the critical thinking skills it taught me. I hope others continue to recognize statistics value in academia and are curious of it.
But I don't think it's just like a statistician shortage. It's that most researchers are rushing to get as many papers as possible published (which, yeah, is arguaby an 'economic' _incentive_ to do statistics sloppily; but mostly I mean they don't feel they have _time_ to do it right), and most universities are trying to cut internally funded research budgets (meaning no money to pay all these extra statisticians, even if they were trained).
It's a "market" problem with how the work of science is actually materially rewarded and sustained. Papers, papers, papers. (Which for that matter -- a good properly trained statistician ought to have the same academic status as the researchers, but you don't get tenure by helping someone else analyze their research...)
Change needs to start at the top, otherwise the reviewers will not understand what they are reviewing, as silly as that sounds.
Couple this with the scientist going to the statistican basically saying "what can I claim with some kind of plausibility", rather than being indifferent to whether the result is interesting or groundbreaking or would make a good clickbait headline, and it's hard to figure out what's true.
It was clear to me that to do the job properly I would have to worry about every thing. And I couldn't trust any of the statisticians to do that well for me, as none of them were aware of all the things.
I suspect this is only done well in extremely narrow domains - maybe nuclear physics (e.g. CERN)? Where everyone present is extremely well educated about the statistics - not as a separate discipline, but as necessary background understanding to do non-statistical jobs too.