Potemkin Data Science (2020)
mcorrell.medium.com
mcorrell.medium.com
Nevertheless, we currently want to be ML people, more or less. It's sexy to download __some__ dataset and develop a new model architecture. It's not really sexy to collect good data - not just any data, but good data.
Now, coming from academia, I think this is just a normal thing. It happens in every discipline in some way. In academic fields that are driven by observational data, data collection is usually a difficult and tedious task. Developing theories, methods and, of course, new results is much more interesting. And rewarding!
Look at economics and Piketty's work: His main contribution was to collect data on wealth. Nobody really cares about his models much. However, for some reason, that data simply did not exist before. And it's now valuable for hundreds of different projects.
As a data scientist, one needs to think about data, first and foremost. One needs to spend most time on collection, on measurement errors, on selection effects, on identification, censoring and all that stuff that nobody likes to think about.
Fitting the next best function approximator to the data is the most fun, but probably not the most valuable thing a data scientist can do.
From that standpoint, I think data scientist is a proper job that is probably needed in every company in the future. However, no coursera course on the latest ML package will make you a valuable data scientists.
I think data scientists who are expert statisticians, survey and experiment designers, causal inference gurus and combine that with proper CS techniques will be valuable.
I think you need to have a well developed software engineering skillset. You need to have some experience with database systems and query optimization. Of course you need a strong background in statistics and mathematics. Often these skillsets are split across teams and departments, which tends to slow the process down. It takes a long time to develop these skills, but if you have them, you are very valuable.
Many DS and sometimes companies make a mistake of undervaluing the software engineering skillset. However, if your data comes from the real world, it will be messy, it will need to be organized. If you don't know how to do that, then you aren't going to build your models. If you are bad at querying your data, then you are going to build your models much more slowly.
1. Data science is often low latency work, running investigations to provide insight for strategic decisions. This is more of a political/comms role - and the cadence/requirements of the role tend to be counter to the long periods of uninterrupted focus that an engineer needs.
2. DE work is foundational, but once it's built, you probably want someone who is fast at adhoc analysis. (Unless you want to push into production - but that's a 3rd skillset, IMO)
3. It's hard to be both user and designer; having two brains has saved me from many mistakes related to bias and assumptions that I make. The DS can function almost as a product manager, synthesizing information and focusing on deliverables (the "what"). The DE can focus on the how.
Would love to hear your thoughts - this is an area that I'm very interested in.
However I think that's a somewhat different issue from the one under discussion here. The problem here is that data scientists can end up wasting time trying to apply fancy techniques, when the real impact would come from basic data analysis using old-school statistics.
Interestingly, I have never encountered this in my personal experience. I don't personally know any data scientist who would rather use in neural network win something simpler would suffice.
Also, I disagree with the idea that data scientists don't need long periods of concentration. If anything, data science is more of a "creative" enterprise, whereas coding tends to be somewhat methodical and mechanical. For me at least, that means long periods of time spent trying different things and contemplating, which requires just as much deep focus as writing a complicated piece of software.
1. This really depends on the DS role at a company. You are describing a company where DS is highly valued and has a lot of power. They probably have representation with a VP or in the C suite. Some companies are like this.
2. DE should be foundational, and they should also be able to push to production. You might have another team which works on setting up new platform environments, but DE better be comfortable pushing and debugging issues in production. It's helpful for DE to be able to perform exploratory analysis on a dataset. It's easy to introduce bugs, if DE can catch this, then it really improves the velocity. If they can't do this, then you need to wait for DS or somebody to act like QA for a dataset. You can find cheaper DEs who are good at shoveling data around, but they don't know what a probability distribution is.
3. It's helpful for DS to have some domain knowledge. If they have that, and DS is highly valued at the company, then it makes sense for the DS role to contain a Product Management component.
I'm glad you are interested in my thoughts on this. If you have any more follow up questions, feel free to ask.
I think you are right that there are organizational trade-offs, especially in day-to-day operations.
However, I would claim that if DE and DS are divorced enough, there is essentially no way that DS work can be done correctly and it would be up to chance whether the DS division in your firm actually does useful work, or does all the things that the article bemoans.
Much frustration with Data Science (also as indicated in the article) arise because DS is seen as an ad-hoc creative enterprise to squeeze "insights" out of an existing data-repository and pipeline. Instead, I'd argue, that anything that touches data needs to be handled by both a DE and a DS (not necessarily the same person).
There are fundamental reasons for this. If you really want to draw insights from data, then you can not simply take some data set and go at it. From a standpoint of statistical identification, data and model go hand in hand. It's not a model that is identified, it's a model on a specific statistical process (that results in a set of data). And frequently, thinking about how to generate and process the data is much more important than the choice of which model to run.
In other words, DS need to be involved from the moment the data starts to exist as such, in particular in the moment the data is collected or generated.Otherwise, DS are working blindfolded - which is really the norm right now.
Think of it this way: if the DS process in your company starts with the tabular data that goes into the model, then there is essentially no way that the process of creating a statistical model can be right. If the outcome works, it is as much chance as it is skill.
If the work of DS is defined by starting with some mysterious yet usable set of input data, collected somewhere else "in the machine" and squeezed together for the DS's convenience, then I highly doubt that DS will be a popular job description in the next ten years!
Again, this does not mean DS and DE should be the same person. But since many points about data are as important as they are subtle, an organizational disconnect is at least risky.
Edit:
Currently the endorsed process is to build a model, eventually be surprised by "data drift" and then rebuild everything.
In my view, this is all backwards. Sure, data changes and one needs to iterate. But this is something the DS team should have been aware of beforehand (in most cases). What I mean here is that the surprise factor conclusively shows that no one really thought about crucial properties of the data while using it. From a statistical standpoint, this casts great doubts on these sort of analyses and models. If you are surprised by data drift, at least to the degree that it seems totally unanticipated, most likely you spent too little time on data and too much time on model.
We all know examples of this. You get "some" data collected on marketing impact on young adults and you find out that doing X does Y. And then later on, the whole thing stops working. NOW you begin an error analysis and find out that data was collected in some semi-random fashion from phone users in the US, whereas now you have new audiences from Twitch (or whatever) in SE Asia. The problem here is not the model. The problem is also not that you need to iterate. The problem is that through the opaque process of data generation no one knew, could know, or cared to know what exactly the properties of the data were.
Right now we are using an ad-hoc approach. Either, because nobody really knows how to characterize the data and what implications for inference this has (for example because statistical issues are dense, poorly communicated and consequently rarely taught in anything ML and DS), or because the organizational disconnect from data collection makes it impossible to reason about these issues.
Data science as a field (and as a job title) is broad and generally splits into
- research
- software engineering and data work
- product/business decision making
With many data science jobs being some blend of the three instead of staying purely in one category. And you can split each category even further. The point is that there is so much breadth that people can have the same title and completely different job descriptions. You need a disclaimer about this pretty much every time you discuss anything data-science related.
The article focuses exclusively on the third category of data science (as does most of the criticism of data science). So to understand the article as criticism towards the field, you need to take two things into account
- what proportion of all data science jobs are predominantly in the third category or involve any influence on business/product decisions (anectodally, not that many)
- how many of those jobs and teams actually exhibit the dysfunctions listed in the article
I don't have any numbers to answer these questions.
Personally, I'm interested in the engineering type of data science jobs (ML engineering, scientific computing and other adjacent roles). I wouldn't want to be in a position where I have to advise on decisions, or come up with justifications for someone's business/product choices, or continually have to come up with excuses for my existence in the company. In my current job search I have a preference for positions titled "software engineer" just to avoid all of this ambiguity. It's also very tempting to switch to being a backend developer or something similar and drop the whole data science thing altogether.
I like your breakdown, and I've observed similar things in my experience as an engineering focused data person! I've had many discussions with my colleagues about how to manage effectively these different blends of roles and skills.
I'm looking for someone for an engineering type of data role right now. Is there a way to get in touch with you about it?
Our product helps companies listen to their customers by unifying natural language feedback across various channels, applying signals using various natural language modeling techniques, then aggregating them to help teams deliver better outcomes using more relevant information.
Hope to hear from you (brandon at frame.ai) :)
edit: forgot to share agreement for your breakdown
In most organizations, data science or analytics will likely play all these Potemkin-ish roles to varying degrees, while also adding real value to the business.
Sure, you may do some "decision laundering" for an executive, but that executive may then go on to help champion data science and develop a data-driven culture because by doing him or her a solid every now and again, you build a relationship. Favor trading may seem dirty or impure to a lot people, but it's also how a lot of great work gets done.
And sure, your org will likely use data science as marketing with a healthy does of McKinsey-inspired corporate BS thrown in. This does not and generally will not preclude the organization from also using data science productively and appropriately.
Anyone who thinks you can rise above petty politics to do some sort of "pure" data science is going to find themselves, frustrated, ineffective and burnt out.
That's not an end in itself, though, is it?
One thing that is worth pointing out is that the deficiency in identifying goals before running after impressive technological complexity as a status symbol is not restricted to just Data Science. I see it all the time with technologies like "Bluetooth". Let's put Bluetooth in everything. We don't know why but it seems impressive so let's just do it.
An unfortunate choice of example in hindsight
> But it’s not true. Indeed, it’s nonsense from start to finish. Jones isn’t a martyr; she’s a myth-peddler. She isn’t a scientist; she’s a fabulist. She’s not a whistleblower; she’s a good old-fashioned confidence trickster.
I think this is one of the greatest flaws of current journalism: By increasing the "feel good value" for its intended audience and confirming their respective biases, an author can completely disenfranchise readers coming from a different point of view.
You and me were both repulsed by the sweeping statements and lack of nuance and, since time is precious and the article looked like yet another propaganda tool, you rejected its value. Rightly so.
Yet in it where description that were correct and pointed to a deeper issue of whom we believe and why.
So by writing for just a section of a possible audience any discussion is impossible.
I won't read the National Review (just this article) and thus their content will be irrelevant to any good discussion I'll have in the future.
- She was a low level GIS developer working on the state dashboard
- After repeatedly posting internal data on personal social media accounts, she was asked to stop and her responsibilities were changed
- She escalated by intentionally sabotaging the rest of the team's work, locking other employees out by removing their security credentials, and by moving data out of the system, crashing the dashboard
- Even after discovery, she was not terminated, and an arrangement was established to have her go through consulting with management
- She then sent an email to everyone working on the dashboard falsely asserting she was being fired for not manipulating the dashboard, which was picked up by the press immediately. In response, she was actually terminated within a couple days.
- Her story, early on, waffled on whether the dashboard was accurate or not, and at times she asserted it was indeed transparent and accurate. Over the last year the story has evolved to the point where she now has a long list of individuals and organizations who apparently have conspired to falsify data.
- She claimed that she was raided by police last year to confiscate her laptop containing data the state didn't want getting out, during which the police waived a gun at her husband and children. However, it turned out she was related entirely unrelated to her work, and they were raiding the house because she had stolen the data of 19,000 employees of the Florida Department of Health from her home, and they traced the IP address there. She also took a video of the encounter in which she waived a sign saying "Biden Hire Me". And bodycam footage from the officers disproved her claims of guns being waived at her family.
- Using her fame and story about being a "whistleblower", she has raised hundreds of thousands of dollars online
- She continues to maintain her own dashboard, but admits she produces it using the same data source that the official state of Florida dashboard does. The difference in what they show is in their methodology for processing the raw data, but other epidemiologists and data scientists claim she clearly overcounts things like deaths and cases due to obvious errors, like counting both PCR positive tests and positive anti-body tests, which would include many vaccinated individuals and double count individuals who take both tests, as well miscalculating "excess deaths".
Apparently she also has long history of felonies/misdemeanor charges including a trial ongoing since 2019 in which she is accused of stalking her previous boyfriend. Hard to say how relevant it is to the dashboard fiasco, but at the very least makes it seem like she is an unstable individual.
At any rate, in addition to the unrelated trials, she is now awaiting trial for the data breach, as a judge signed a warrant for her arrest after reviewing the evidence seized in the aforementioned police raid.
The vast majority of data scientists are terrible at their jobs. They lack fundamental skills on how to extract business value from data and share that value with the organization.
The fact of the matter is the majority of organizations don’t need FANG style data science with esoteric models and pipelines.
It’s very similar to the current software developer trend of over/pre optimizing or resume driven development that is ravaging the industry today.
Somewhere in the same company might be someone making the same as you with a couple of SQL/Qlik courses in their well groomed head and a handful of ironed shirts in their closet. But we will look down at them because dressing well is for sheep and SQL is like, 100 years old.
In all honesty, almost all firms that are hiring consultants to do data science projects (based on the sample I've been exposed to) fall firmly into the situations the article describes. It's all about appearance and justifying existing decisions or plans, and even the suggestion of changing something swamped out of existence by the bureaucratic immune system of these orgs.
On the other hand, I see (especially in advertising but also e.g. fintech and others) cases where an internal data science team really does drive the agenda, and thing like AB testing are used for real in order to make changes that result in more money.
Lastly, the in-house team at a legacy company. Although I have less experience here, the ones I've seen closely mirror "innovation" or transformation groups that are supposed to come up with new initiatives or changes but inevitably are ignored because they have no links to operations.
So over all, the article resonated. As a data scientist , there is meaningful work at companies where data is closely linked to operations, mostly in "new" businesses. As a consultant, I find it interesting to understand legacy businesses and their data, and learn about what potential upside there is, even if I know they won't listen to me.
The more interesting line of argument to support data science being a bullshit job is that all of these lucrative use cases are just a tragedy of the commons. Insurers in aggregate did just fine when pricing was based on actuaries using simple approaches to "square the triangle" and then adjust rates based on rules of thumb, the only reason they can't always do that today is that everyone else has data scientists. And if you don't have a good recommender system, your customer will just buy somewhere else (or "worse", avoid an unneeded impulse buy).
Of course, it's not strictly zero-sum. Insurance pricing is "fairer" now, which is probably a good thing, and data science is enabling all sorts of obviously positive-sum products like wearable health monitoring and car safety features (to say nothing of self-driving eventually). Other use cases like "engagement" optimization are probably negative-sum. But in general my point is that creating millions of dollars' worth of business value per year, as many (most?) data scientists do, doesn't preclude you (or me) from having a bullshit job.
For many roles and positions in a company, high and low, you can tell whether the person is doing a good (or at least competent) job. A sales rep can show leads he/she has followed up or converted, and $ made. A coder can point to functionality created and lines written. A CFO can point to deals done, documents prepared, etc.etc.
With data science, sometimes you just have absolutely no idea whether the person is good or not.
"This problem can't be solved". You just don't know whether it's true or not.
"These are amazing insights". Do they have any practical value that can turn into $?
Data science is from some perspectives the most black box part of a modern organization. And so much hype is put into it, so many people paid sky-high salaries in the belief that it is producing something.
Is it usually? Not sure.
Trying to shoehorn something complicated into a set of broad statements does not help to address the underlying issues as they are not specific enough hence not actionable.
What is more "Data science" seems to be an ambiguous definition. Besides it is intrinsically an afterthought as data does not exist until the company collects it. I would also argue it is "a thing" only in large orgs or globally successful software; small companies do not posses datasets worthy of exploration.
Specifics would help as otherwise it feels like we're talking about technical debt.
Or with a cash flow for a permanent position. Tech workers are used to be in companies with healthy gross margins and revenue where they make sense (theoretically at least). But even it tech I would not be surprised if there are teams of DS/SWE people in Uber or Doordash helping their companies to lose money faster.
I’ve been trying to learn data science and machine learning. However, data science as a field seems to have already fallen out of favor.
(In the same way that "DevOps" is just a fancy name for a "sysadmin with some light scripting skills".)
It's an extension of lean manufacturing to software development, where the focus is on continuously pushing tiny packets of functionality to customers (instead of occasionally pushing huge bundles of features), to react quicker to changing demand and mistaken assumptions.
It is in no way "a sysadmin with scripting skills" -- the name for which, is, by the way, just sysadmin. Sysadmins know how to write Perl and shell and Python and have done so since forever.
> ...the name for which, is, by the way, just sysadmin
No, because some time in the past "IT support" (you know, the guys setting up Windows and Wi-Fi routers in your office) got upgraded in name to "sysadmins", and so the old "sysadmins" who knew Unix and how to code some Perl needed a new, untarnished name. Hence, "DevOps".
Fun times trying to get code you didn't write to compile without knowing any context.
Anyways, that sort of turned me off trying to move into programming and so I went back to mechanical engineering after less than a year. Maybe I should have stuck it out longer.
The thing is that Data department have essentially swallowed the former analytics departments, and many people who have done business intelligence/business analytics now seem to fall into the data science umbrella.
This is part of the reason the term now refers to different things depending on who you talk to or which team they are a part of. Look at job postings, its extremely difficult to understand what the actual job entails these days.
morally ambiguous act of consuming
Focusing on this problem we are discussing, how much can you control the effects of whar you consume other than by controlling how much? You might buy vegan patties in whole foods, but other than the fraction of your money that went to the direct purchase of energy or land for agriculture, the money just gets split between roughly the remainder of the economically contributing population. Thus as long as you consume, you only have partial power over it's good-faith nature, the remainder is controlled by how much the entire population (and businesses) act in good faith in general. If you really want to keep your hands clean then I see no way other than not to spend a single dime more than you absolutely need to for your survival. Everything else is morally ambiguous at best.
And this statement that we absolutely need the economy to grow for continued prosperity is preposterous. It's clearly A method of continuing prosperity, and clearly a method that at least works. But it's not proven to be the only method and we better start inventing new ones that don't put all their money on what's basically a glorified Ponzi scheme that everyone's forced to play.
???
The world is very decidedly not zero sum, you don't need economics to tell you that. Zero-sum games are a particular case.
You might argue that it's not zero sum for practical purposes, but to me it looks very decidedly zero sum in many different dimensions (usable land, co2 release, fresh water, many minerals, list goes on).
Utility isn't measured in amount of land used or CO2 emitted or minerals extracted. This is not a Starcraft game.
Using land better increases the sum. Making energy production more efficient increases the sum. Zero-sum means that any of my gains is someone else's loss, which is not how any human society has ever worked.
A zero-sum world implies that any form of human cooperation that isn't literally war is impossible, which doesn't empirically seem to be the case.
However, that doesn't in any way invalidate the claim that parts of the economy are in fact zero-sum. How large are those zero-sum parts - 10%? 20%? 40%? 80%? Nobody truly knows. Needless to say, the optimal survival strategy in a world of 80% zero-sum is vastly different than in the 10% world.
Also, although there's no data to back it up, I believe the zero-sum games as a proportion of our society are growing, and quite rapidly at that. It's a very worrying trend.
Now, some points.
what economics is a science now
how much can you control the effects of whar you consume other than by controlling how much
You might buy vegan patties…
you only have partial power over it's good-faith nature
If you really want to keep your hands clean
more than you absolutely need to for your survival
we absolutely need the economy to grow for continued prosperity
preposterous
It's clearly A method
a glorified Ponzi scheme
This doesn't seem right to me. If I buy food from the grocery store and eat it, does that shrink the economy? If I enjoy a meal at a restaurant and tip the server, does that shrink the economy? I think both of those actions grow the economy. Both of these actions would grow the country's GDP.
Even though I don't agree with the reasoning, it is interesting. Essentially since a material has been used, and is no longer available, now the whole economy is poorer. There are different ways of measuring wealth, and you seem to be endorsing the amount of materials that an economy has determines its total wealth.
Now, insofar as the goal of the economy is people's enjoyment of stuff, it served its goal, which is great. But don't think that by consuming the food you did everyone else (the economy) any good - you didn't. You personally benefitted, which is great, and used up the resources - people's time and effort as well as materials that went into it.
A better illustration would be you commissioning a yacht for yourself, using it recklessly and sinking it (with its value going to 0 - i.e. you consuming all of the value). How did the economy benefit? Engineers that could have worked on better problems, e.g. automating labour, instead spent their time on your yacht. So did workers. So did all the people to procure the materials. And now all of that is at the bottom of the ocean, benefitting nobody. But you had your fun while consuming this product.
GDP is a lousy measure of a success of an economy. The point of an economy is to serve the needs of the people - sometimes it seems that some people believe it is the other way round, and people exist to 'grow the GDP'. If people are happy with consuming less, they can work less - what's so bad about that? Or they can invest instead of consuming - that is, work hard now so that they can work less in the future (or have more/better stuff). It is the investment, production and innovation that grows the economy in a meaningful sense (or getting access to cheaper labour, which does not seem very ethical).
Encouraging consumption under the guise of benefitting the economy is absurd.