Why Are Data Science Leaders Running for the Exit?
linkedin.com
linkedin.com
This blog post would be more compelling if there were citations or even general context backing up the authors pretty wild claims ("the vast majority (we are talking 80-90%) want to leave their current job", "failure rate of data science teams is over 90% right now", "the top 5 data scientists I have ever worked with, only 1 had a PhD").
As a data scientist working for a relatively well known startup in the Bay Area, I don't dispute that there are problems in this industry. There are many under-/un-qualified managers running around (I don't know if that's a data science specific problem, or if it also applies to much of engineering as well). There are ICs and teams that have trouble producing industry-applicable work because they're too academically focused. There are also business-driven teams that have trouble producing quality/reproducible work because they're too business focused.
That said, the claim of 90% failure is ridiculous on its face. Having another MBA put together a powerpoint presentation about a company's data strategy isn't going to suddenly solve the actual issues in data science.
In my opinion, I feel like we are at peak "!@#@ data science! @#$@", and there will naturally be some deflation and correction in expectations just like any other occupation that comes out of the woodwork and is the 'sexiest job of the _____'. It's just the way it is. The article author seems to be carving himself out a niche to capitalize on this (which is fine).
Ironically, I assumed he was talking about NYC. Apparently he's actually in Minneapolis
"I made a huge difference for the business by being thoughtful about feature selection and then applying a bog-standard regression method" isn't appealing to the academic mindset.
So, what I mean is that people are looking for something where they can apply one team's work and use it across many teams. The more the better. The idea is that you don't have to have many people being thoughtful producing methods, you just need many working hard applying them. If there is a better term for that, I'm game to use it.
This is a problem no matter what line of work you're in.
I considered flagging the article but decided against it after reading your comment, which I hope remains visible.
It is unsupported in the article, but on its face, isn't a 90% failure rate about right for bleeding edge tech efforts?
Ask a databases person and they will tell you about ETLs and relational joins. Ask a statistician and you will hear about estimators and the bias-variance tradeoff. Ask a social scientist and they will tell you about experiments and causal inference. Ask a CS theoretician and they will mention streaming algorithms and probabilistic guarantees. Ask a machine learning person and you will probably hear a lot about prediction and stochastic gradient descent.
The approach that will breed the best practitioners is an interdisciplinary one; a great example is UBC's program: https://masterdatascience.science.ubc.ca/program/courses. And also future positions that separate the roles of data janitor, data analyst, machine learning engineer, and so on. There is a huge difference between what Facebook's Core Data Science team does and what most companies call "data science", which is sitting on a huge pile of poorly aggregated data and having little idea what to do with it.
Most importantly, data science has to be thoughtful and part of the engineering process, and can't be done after the fact with digital trace data. It's a lot easier to run experiments than it is to do statistical gymnastics with observational data.
PhD's are notoriously (and understandably) finicky about doing non-research work when the job is sold as research, that's why you have to be sure you really need them. That issue has been around since long before data science in SaaS companies was a thing, it's been around since before the internet was a thing.
Assuming that there really is an exodus problem, PhDs being unprepared to lead teams and do non-data-science work is one theory that seems reasonable, but it definitely isn't covering all possibilities.
Having been in a couple of companies hiring data scientists recently, I personally think the big problem is expectation for data science in general, more than who's been hired to do the job. The couple of companies I've worked with, anecdotally, seem to expect "data science" to uncover huge missing profits, expose gaping inefficiencies, and reveal new revenue streams the business people didn't see. And when there turns out to be only minor ways to streamline things, the team of data scientists need something else to do.
for most, Big Data has been a sham, but it was very profitable for cloud vendors
I've worked in many Fortune 500's with "data science" or "big data" teams. These are well staffed, very expensive teams that have large budgets for pricey hardware (sometimes on-prem, sometimes in the cloud)
I have never seen one of these teams produce insights or actionable intelligence valued anywhere near their cost. I mean not even close. Usually it is a tremendous money fire. (Also, before the pitchforks come out, I'm sure there are places where the data science team is a profitable department, but it's not the norm, not by any stretch)
Part of the problem is the business doesn't know what questions to ask. Part of the problem is the technology itself. Spark streaming, Hadoop, and all the other tools really aren't very good (very good being defined by helping businesses answer burning questions in a reliable and timely manner)
The most valuable data insights I've seen come from purpose built analytics tools using simple storage backends (RDBMS, elasticsearch etc) where the person running the team is a domain expert, not a "data scientist".
I've seen some customers of ours being quoted six and seven figure prices by their internal data science teams to essentially slurp and transform a handful of SQL tables. It's the kind of thing that shouldn't cost six or seven hours to write the code for.
They don't wanna do it so they give a ridiculous estimate
Then, and only then, should you look at your service levels and determine if there is a need for some sort of Hadoop or Spark infrastructure.
Unless it involves customers and clients, in which case it's actually a sneaky liability... At least in terms of PR, if not legality.
The second, however, is more subtle but often more damaging: people often assume that the data they have is the right data or complete so the drunkard's lampost problem is easy to fall into and people might not realize it as quickly because, hey, those numbers were based on so much data it took a day to run the query! Web analytics is notorious for that — people would make statements about performance, browser support, etc. from a bunch of log files and miss that this was skewed by bot traffic, confuse their server's response time as being a reasonable proxy for the user's perceived load-time, etc.
I say this from personal experience. The major risk here is that Data Scientists will constantly get pulled in as resources to support business fires and high visibility projects that should be handled by other people.
Otherwise what happens is this: "I know you are working on that Data Science project that is scoped for 4 weeks, but can you do this Business Analyst thing the insert exec name here asked for by this Friday?"
Data Scientists need to be shielded from that day to day analyst stuff, otherwise they'll never be able to do their jobs. One of the ways to do this is to put them in IT in a business facing consulting role. IT is typically (not always) is more focused on a proper solution, not day to day triage. So they can filter out projects that aren't appropriate.
The data scientists that I have seen that are successful and bring the most value and seem to have the most satisfaction are those that spend time actually doing analysis and provide deep insights into the problems they are looking in to. The answers are the easy part - asking the right question is the difficult part.
I am reminded of climate-change skeptic articles that say “hey these scientists know nothing. See today it’s 10 below zero. Common sense! No global warming.”
And yes, we should all appreciate not just the named author of the work, but all of the work that goes into it. This is true of all works, not just studies.
That is a recipe for disaster - non-technical people leading technical people. Maybe we should hire an MBA as VP of Engineering too?
And in many other cases, Media has created a myth of Data Scientists as Magicians, hence a lot of people expect a lot of unknowns from Data Scientists while a lot of people become Data Scientists always expecting to create unknowns and magics which is not going to yield any good better than a mere Exploratory Data Analysis based Insight from Microsoft Excel
I got frustrated enough to where I almost quit on the spot a few weeks ago. Frustrated enough to b on the verge of tears and punches because I felt I'd wasted the past two years of my life. I worked to get myself transferred from my leadership role to, ostensibly, a more thought leadership type role which should happen in next few weeks.
However, if I don't see significant changes in January, February, and March then I'm leaving. Honestly, I'll probably leave anyway because I think I can grow my skillset, work on better problems, and set my future up better somewhere else.
What I hate about this situation is that I used to be the biggest supporter this company had. But years of this difficulty is enough.
The trouble is of course we don't have a good name for that role, so it's all "data science".
DATA science is more like quant consulting.It is actually a subdiscipline in management science in my opinion.
The key role of a data scientist in non SV company is usually 6-fold. First understand the business problem thoroughly. Get the people,process and value stack nailed.
Second and this is the most imp part. Figure out how to frame the business problem. Framing isnt as linear as it looks. But it helps save a ton of time and money if you front load time in framing problem. Substep here is to frame it as an "information" or data problem.
Third articulate a strategy to gather data or information. Inhouse data available? Web scrape data, buy data, build generative models. Youd be suprised how much ML work happens in this step
Fourth shape the information to a format that can be analysed.
Fifth, study the hell out of this dataset- the PHD angle comes here. Very few people are systematically trained to study data with rigor and be humble and honest about their findings.
Lastly connect the dots between insights and business problem at hand.too often this stops with bar graphs and some scatter plots. Thats just lazy. You need to really take ownership of the problem and educate business why the solution will help and be honest about ehat it will take to get it to work. The people, process,value stack in line 1 kicks in here in recommendation and action.
A bonus point. Put some kpi to track how well your suggestions are working.
Real data science is super super hard. Its like finding a real nugget of diamond in dust.
Data science is a way thinking. Its a business culture to be honest.
but in the very few ds interviews I had, I was tanked as soon I got asked questions like: whats the formula for a T test. I know what the test is, and when its not appropriate to apply it, but I don't memorize those kinds of formulas. the field is just too big to do that
Were you making it past the initial stage/screen interviews where every question is a buzz-word test? Were you ever interviewed by people who you'd actually work with and know what their team needs? What geographical area do you reside roughly, if I may ask?
This position was in NYC.
Experimental design (statistics) is important, but data science is a team sport. The people specialized in statistics may not be able to code up a deep net, and vice versa.
Computer scientists make great data scientists, especially if you know data infra. If I were to start a team from scratch, I'd focus on software engineers and physicists. Most are willing to pick up some ML to broaden their skills, and the ability to go end-to-end from conception to production is invaluable.
The real issue is setting expectations with those on the outside trying to get in - a 3-month boot camp with no prior programming or statistics knowledge is just not going to get you to the level of competence required for this type of career. All of these MOOCs for ML/AI/DS are bullshit if you haven't the foundation underneath, which, based off of those people I know who've taken those courses, doesn't exist for many MOOCers.
The fact of the matter is that most companies don't need "artificial intelligence"; they need intelligence - people with domain and statistical knowledge and enough programming skills to be dangerous. As cool as some AI/ML tech is, you could get so far as a data scientist with great knowledge of a few statistical approaches and above-average programming skills.
You're right in one sense, though: there are a lot of data science practices/applications that aren't worth the value they provide. There are others that are worth exponentially more than their operating costs. As with most things in statistics, whether data science is effective "depends."
Others have noted the lack of data coming from a data strategist. That is the feature, not the flaw. Data isn't necessary in these people's worlds. They spout platitudes about data, laud it when it supports their world view, and ignore it when it doesn't. People that buy the world that this guy is selling are suckers.
On an unrelated note I don't really buy into the idea of data science as a distinct thing from computer science.
Depends. The interview process that is the fad these days selects for assembly workers. Organizations using that process are doing so to find people who aren't curious about broadening their knowledge and experience, but to find people highly competent at repeatedly doing their CS curriculum over and over.
> On an unrelated note I don't really buy into the idea of data science as a distinct thing from computer science
How is it not distinct? Data science is closer to a physical engineering discipline than an applied math one. The tools of the trade might be enhanced by application of CS, but the two are quite distinct.
And why are they doing this? Why don't they just buy an existing solution from another vendor, why build your own solution?
There's a definite place for these workers.
You are unlikely to ever achieve a position in industry that has the same freedom of investigation and time that you experienced as a graduate student, let alone the holy grail of a self-funded post-doc or the like. On the other hand, you won't achieve that as faculty either. In industry you're going to end up spending a lot of time on activities and meetings that don't relate to what you now consider "what you do" (e.g. ETL, domain interview, co-ordination etc.). Your success and happiness in this will depend a lot on whether you consider those things a waste of time taking you away from what you "really do", or an important part of how you do such activities in larger groups.
On the other hand, data science has a similar problem as software programming; failing to know how to find or produce good managers for it, companies tend to promote from the ranks of their best individual contributors. This can work well with enough support and mentorship (assuming the candidate wants to do it) but it can be a disaster without a plan and that support.
which explains my advisor's "enjoy it while you still can!" attitude whenever we talk of such things :)
Academics are rewarded for building complicated things, but it often takes graduate PhDs entering industry a long time to learn about quality of execution. I think that's the fundamental tension between business and academia.
This statement "The more complicated a model is, the less well you can execute it" is fishy though. The typical problem in practice with complicated models is the data quality and quantity to support it, not the implementation or interpretation. I suppose here we also need to define "complexity" - I am using this to mean roughly number of parameters.
A full-time data engineer should probably have been hired.
I wonder if this leads to an analogous discussion as http://wiki.c2.com/?ArchitectsDontCode.
You told me last year that one data scientist is gonna solve all my problems. Now you're telling me I need to babysit him with five data engineers, because he is not going to get his hands dirty with data??
Who would sign up if they told you up front that your main contributions would be shuffling email, filling out TPS reports, attending pointless meetings, and being roadblocked every time you tried to automate, eliminate, or improve that drudgery? You can smash your head against the wall trying to make things better, but all that ends up doing is pissing you off and giving you a headache.
The biggest problem can be summarized like this: If you want to run your business differently, let the new people you hire do something different. If you want to do the same old thing and just act you are doing something differently, just teach the old people to talk like the new people.
I've got plenty of anecdotes from the year, but here's a simple one. The division had an existing "Research and Analytics" group that used linear regression in Excel to develop models. They would develop models on awful (read: highly-multicollinear) data with 100 rows and get "R-squared of .98." Large long-term business decisions were made on these models.
Over the course of the year I fought this constantly. Why do you need more/better data when the other team is okay with the current data? Why are you struggling to get your R-squared to >.90 (output of any type of model had to be R-squared) when the other team can do it so easily? Why do you need to speak to so many people to understand the data better? What I realized over the course of the year was that nobody actually cared about the models unless they matched the "gut" of senior management. So it was easy for the other team to p-hack until they got a slidedeck people would like.
I think there are plenty of problems that lend themselves to good Data Science. These are ones where you can properly identify what a prediction or better understanding of data (e.g. variable importance, coefficient confidence intervals) means in terms of resources (e.g. people, money, time). You can also identify what it means to not have that data insight or model.
There are also plenty of business areas where the link between resource investment and long term success is less clear (e.g. Marketing, Sales Incentives) and more gut and smart work than science. A lot of people are trying to clear up that grey with data science, but I go back to what I said before: you need to let people think and try different things over time to see the value of those experiments.
Comically enough, the Research and Analysis team picked up all the lingo we used and became a huge success: work the old way, talk in the new way. Management decided their Data Science adventure was a failed experiment and we just hadn't lived up to the hype.
I joined a non-tech company as their first data scientist, and am now tasked with building up the data science efforts of the company. Your comment is really ringing true, though I've found the company here to be generally great to work with in terms of their flexibility. Funny enough, it took me months of working with the IT department to get a VM with 8-cores and 64 GB of RAM, or the ability to spin up AWS instances (none of the analysts at the company had ever asked for that). I'm happy with it though, it's a fascinating challenge and I'll admit at times I feel a bit in over my head.
In my experience, many companies are wanting to hire a lot more data scientists (and paying them six-figure salaries to boot), when what they really are looking for are what has traditionally been called a "data analyst" position. I.E., the entire sum of the job is Excel reporting and basic statistics, but instead wrapped in some "Machine Learning" marketing language.
As opposed to more long-term focus on what business problems can be solved by carefully cleaning data, building models, testing predictive power, and then deploying these models into production. Which obviously takes a lot more time, a lot more effort focusing on a single problem, and doesn't have nearly as many immediate deliverables that higher-ups can open their Outlook inbox and see. Naturally they then wonder why they are paying such a premium for someone who doesn't bring as many deliverables to the table as their 60-70k analysts. And then the data scientist wonders why he's spending most of his time sending Excel attachments. Definitely more of a problem in non-tech companies.
Most businesses think small. I'm not even sure it's about risk/reward, either. I think people really don't have vision especially when it comes to technology and more specifically data.
The next generation is coming, though.
Having some expertise with BI applications (e.g. Tableau) has been a major strength for showing value.
100% Agree.
Some of the execs I reported into were obsessed with the idea of being a startup within a large organization, and part of this was the assumption that one of the things startups did was use data better than big companies. To them, there was some hidden value in their data that their Business Analysts did not have the skills to unlock. Data Scientists are perceived as the magic key to unlock this value, even when what the business is asking for doesn't make sense or isn't feasible in the short-term.
One of the small ways we delivered value was to write small ETL scripts that fed into Tableau dashboards. People loved the Dashboards because it cut down on so much Excel stuff that was done weekly/monthly/quarterly, etc. The company did have a large IT org with a BI team, but they were always busy and took months to do anything. So for us to do ETL+Tableau in a few days was seen as a miracle. Plus, Tableau has an Excel export :)
Of course, ETL+Tableau wasn't enough to retain the team.
I tried warning people off of this field given that they would be doing sql query monkey work and drawing simple line charts...most definitely not an upgrade from software development like it was sold as
Doesn't invalidate your point, but the majority of jobs under the very broad 'data science' label just aren't super interesting after all. Guess you just have to be careful to examine exactly what a specific 'data science' position entails.
Any decent Comp. Sci. University will have enough math and statistics.
if a cloud vendor deploys a tool that derives some canned results very quickly at the push of a button, business leaders will prefer those to magic deep queries that take a weekend to run
While I certainly can see overhype in the space leading to team failure (and I have indeed seen that), I'm just surprised that the 10% of teams I've been on were so successful by comparison.
But one of the nasty things I see happening in the space is that often times really great data science folks, in studying a space, come up with a way to invalidate a lot of the business models. If a major player adopted these, there is a chance of success, but it's also a major risk (as it basically unseats the rest of the business).
As such, I think there is a lot of pressure for good data scientists to end up in small teams leading what we'd call 'highly disruptive' businesses. We see this in the finance sector (where it's least likely to succeed) all the time.
It seems like 90% of these guys are buzzword wankers who jumped onto the "Big $$$" bandwagon. Same type of person who put "PHP Wizard" on their resume in 2005.
Good riddance.
I'm sure, of course, that quite a few of them are decent, intelligent people producing real value at real companies. But the ones I've dealt with? Nah.
I think many of them are PhD students who are leaving academia, voluntarily or not, and data science is a logical first job.
Like I said, I'm sure "data science" as a science is great, and I'm sure there's plenty of "data scientists" that I would love to sit down and talk with and learn from. I just have never met any.
Alot of times, it means "the guy who does crap in SPSS or needs Tableau".