We don't need data scientists, we need data engineers

The best quants are 1/3 statistician, 1/3 developer and 1/3 trader, in my view.

I'm in industrial automation, but it's much the same. Projects where someone developed a strategy but has never been involved in the details of a machine are doomed to failure (or at best to be unreliable and producing low quality parts). Projects built by machine fabricators are over-engineered, frequently late, and sometimes unprofitable, but damn if they don't work well.

The main trouble, I think, is that when a shiny new contraption is brought to the king, it's too often the magicians doing the talking - whether they're speaking words of power or Common, their job is to talk. Meanwhile, the blacksmith is probably busy at in his workshop some ornate scroll work for the next thing, or repairing the previous gizmo, because he'd rather be hammering away at his anvil than talking.

The higher you go in an org chart, the fewer the number of people who understand the work their company actually does, and the more voices you have between the workers and the decision-makers to take some of the credit for work as it passes up the chain.

1) They probably aren't going to produce good models since they're not sensitive to data nuances, but now they've taken over ALL the modeling work.

2) They bring down the job satisfaction of everyone else on the team who would like to be doing at least some modeling.

3) They're sucking up the prestige that should be distributed over the entire team and management thinks they should be paid more for work that it turns out everybody thinks is more fun anyway.

My number one advice to entry level data scientists is to not be this guy. Don't give your interviewers the impression that you won't do your own engineering work because they won't want someone who brings negative value to the team.

My god, this. These people make me bonkers. Especially because I feel like I have a bit of this tendency myself, the desire just to think big thoughts and do no actual work. Happily, I long ago learned that ideas were approximately worthless without labor, and that I anyway had much better ideas when laboring because it forced me to engage with the details.

And yes, those people can poison a team. My best working experiences have all been with people who a) all valued actual work and b) believed that everybody could have good ideas.

Inevitably if you treat a job role as a support role, you'll attract weaker individuals into that role then you would get if it wasn't considered a support role. The problem with Science oriented teams is that all roles other than the science role morph into science support roles over time. The same pattern used to occur with Engineers and QA, or Engineers and ops.

Magicians will be magicians, always hustling (bullshitting), but they will never have the value and job security of the blacksmith. The blacksmith can see the fruits of her own labour, whilst the magician must lie to herself and others in order to claim the blacksmith's value as her own.

If the blacksmith is good enough, she will earn the trust of management and management may consult the blacksmith in the selection of magicians. Management may ask the blacksmith to interview magicians and seek her advice on the final hiring decision.

The blacksmith may not carry the "prestige" of the hustling, bullshitting magician but she can command a high salary and dictate her own working conditions. This is only if management understands her value. What the magician thinks of the blacksmith is irrelevant.

Reliable blacksmiths are hard to find. Magicians are a dime-a-dozen.

Is this because it's easier (obviously) to teach a quant engineering than it is to teach an engineer quant finance? Or rather because it's expected now that traders will become the bridge between researcher models and implementation, and engineers will simply provide the underlying infrastructure to power these implementations?

The (repetitive) blacksmith role is not an interesting one, digital revolution needs to come into place. Architects that build tools, self service systems are much more interesting.

Right now our product has accumulated a lot of technical debt on the data validation side because data scientists designed the test code in a way that dramatically slows the development process.

Not to disparage the amazing data scientists I've worked with, but I've been on teams where this is very much the approach to operationalizing models. It's basically, "Here's the sklearn model and some fragile featurization scripts we built. Can you take this to prod ASAP?"

The problem I've seen is that DS & DE teams were in different parts of the org and had their own sprints that were in no way connected. So they kept chucking models over the wall and we kept trying to faithfully operationalize. Once we convinced leadership that we had to collaborate from the get-go, things went a whole lot better. It also improved the working relationship of engineers and scientists.

I learned a hell of a lot from the scientists; they learned how to write better code. They also learned what code they didn't need to write because I could do it faster or better than them, leaving them to focus on more important things. It was pretty amazing to find what manual processes they would setup in lieu of proper (or even any) engineering support. Again, these are amazingly smart people, but they were being square-pegged into a lot of round-hole engineering tasks.

Now, the much more frustrating issue I had was being in a very data-heavy organization and being told by a distinguished engineer (my skip-level) plus my direct manager that, "data engineering isn't a real discipline." I left that org very shortly thereafter.

I feel seen. At a previous job, our output after some cleaning and transforming was a pg_dump for the data scientists to load. We had little visibility of what they did to that database once they got it.

We've been through this with installer writers, database admins, test automation, operations people, and now 'devops' people who were supposed to be the answer to these problems. It never stops.

Reading this thread has made me realize just how lucky I am to work very closely with strong a very strong Data Scientist, who is complemented by a very strong Data Engineer. Conversations with the Data Scientist are always about strategy, product alignment, and ensuring we're optimizing what we build for learning. The Data Engineer works very closely to ensure we're actually capturing the data we think we are, getting it to analysis systems, and making sure those data pipelines stay healthy.

This matches my observations as well. I'm an engineer (The non software kind) at an Industrial plant, I have noticed similar in my involvement with data scientists.

I think in a lot of cases it needs to be acknowledged that data scientists are not domain subject matter experts. Very often the data scientists we have worked with lack knowledge I take for granted as an engineer such as knowledge of basic chemistry, physics etc. I can sanity check plant data almost instantly. For example I will know if a material reacts in an endothermic or exothermic manner and can verify that its effect on a temperature prediction model make sense.

As a result I often feel like Data Scientists are not empowered to bring their full expertise to bear, they don't understand our process fully and lack a lot "engineering" knowledge to make value added inferences about what their models are demonstrating. Often they can deliver a model and show that a particular term is significant but they have a very shallow understanding of what the term actually represents and can't provide concrete recommendations as to how we could modify our plant to benefit from what their model is demonstrating.

Sometimes I feel like we need an additional translator sitting between who can speak both "Data science" and "Engineer" I don't think this is quite what "Data Engineer" as suggested by parent article is but possibly the role could be expanded to incorporate this.

And, their production alleged "machine learning" (it's pretty much standard linear regression, but calling it ML is sexy) systems are slow motion train-wrecks. If the string and duct tape holds, then it works, but it's unfortunately continually breaking.

Hell, in Slack, I watch their data scientists continuously wrestle with how to actually make their Jupyter notebooks work in production.

Whereas my company has far more data engineers than data scientists. The plan from higher up the corporate food chain was always that we'd give them our data to do their data science voodoo on, but we ended up getting a few data scientists of our own for specific projects.

So, we focused on ensuring our data stream was reliable, consistent and sufficiently timely, for them to work on. But as soon as it hits their systems, it's a forest fire of hacks upon hacks, which inevitably break.

In the end, we had to send in our data engineers to stabilise their flagship "real time reporting" product that corporate was so amped about.

So yeah, I think that there's probably a happy ratio of data scientists to data engineers, of about, say, 1:5 or 1:10, because the maths generally scales O(1), it's the beauty of maths, but the actual engineering to get clean data delivered timely without breaking anything scales very differently indeed.

As application engineers build increasingly "stateless" code (e.g. pure functions, serverless deployments, etc), that state gets pushed elsewhere. Someone has to manage the queues, file versions/locations, logs, databases, configurations and so on. That is all "data".

State management is a tricky problem even in a single-threaded application. It's doubly so in distributed systems, where state can be inconsistent between all the moving pieces. This is the source of endless data integrity issues. I think data engineering is a great way to get some exposure to all of this.

https://managingml.substack.com/p/the-myth-that-machine-lear...

It has been my experience too. Basically, ML / DS engineers are thrown under the bus for being poor general software engineers, but in practice it’s totally the opposite.

This can be tough because there could be a lot in that skill set. You can't realistically expect someone to have solid knowledge of statistics including specialising in the sub-field and type of algorithms that your product needs, and also be able to write good code and act as a developer, and also have solid knowledge of all the tools for data streaming/processing/ETL. There is a point at which you're just stretching yourself too thin if you try to do all of these at once.

Of course, stuff like knowing how to interact with a database or employing good software development practices should be a very basic prerequisite and some scientists certainly shift things too far in the other direction and use their academic knowledge as an excuse to write poor code and not learn new tools.

I guess what I'm trying to say is that they are distinct skills but you still need all of them to some extent and striking the correct balance in one's skillset is really difficult.

The actual process of collecting, aggregating, cleaning and verifying data is a hugely important skill, and not one I've really seen typical data scientists possess.

If there is a question mark here - it's really how much value are we deriving from all of these data people?

Where is all the ML that's changing our lives? Search, Alexa and TikTok, I can see it.

In the future obviously vision systems for autonomous cars etc..

But I'm really wary about the heavily decreasing marginal returns after that.

It will surely change the world, but I think in specific areas. Most of the entire field seems like an optimization on something rather than anything new.

Washing Machines feed up immense amount of labour and toil. Alexa telling me the weather is not.

Everyone is buzzing about the latter, and few even realize what is the former.

E.g. getting a model from 0% accuracy to 70% accuracy might be a couple Pytorch library calls that any dummy who watched Andrew Ng’s course can do. But getting that same model from 70% to 75% accuracy might be deeply mathematical and require the latest and greatest mathmeticians, statisticians.

But in this hypothetical example, an engineer who stands up the 70% model and keeps it running 99.9% of the time, with high uptime; is more valuable to the bottom line of the business than the 75% accuracy model hacked together with scripts with 50% uptime.

Data scientists like to quip that 80% of the job is data cleaning, with the remaining 20% divided up arbitrarily among other tasks as suited the joke. In some shops nowadays, it's more like 45% data cleaning, 45% data engineering/ops/programming just trying to make your results available to the rest of your organization, and 10% research.

If I can spend less time learning/doing software engineering and devops and more time doing actual data science, that's great. At a previous job, my team was clamoring for more data engineer hiring, and part of the reason our projects were slipping and starting to fail was lack of data engineering support. Our tooling was shit, our processes were shit, our code was shit, and access to (and trust of) our data sources was especially wet and stinky shit.

It made the daily work of doing data science a miserable slog of ad-hoc duct-tape solutions, and it contributed to us being generally ineffective as a team.

All of this would have been fixed if we had one competent data engineer with some actual real-world data/ML engineering experience and good communication/advocacy skills. Let alone two or three!

Those who have been fed pristine data without having to undergo the trials and tribulations of actually having to collect the data have missed a crucial part of scientific training. Like you, I find this lack of rigour is rather common among data scientists. Not all, but quite a few.

No good data scientist should ever expect data to be pristine. And a good data scientist, even if they don't have quite the engineering chops necessary to build a production-quality ETL, should know enough about the process to help guide it. If they aren't a part of that process, they're not being a good DS. They can't expect someone not involved with their problem to know what tradeoffs to make, and if they don't know exactly how their data went from raw form to the ETL-ed form, they're probably going to make bad assumptions, and those assumptions may very well make their architected solution a complete pile of garbage. Not to mention, how can a DS offer suggestions for solutions if they aren't deeply familiar with the raw data that's available?

To me, a good data scientist should, at bare minimum, have several skills.

* They should first and foremost (but not solely) be an in house expert in statistics and machine learning to know what can be done with data, and what can't be done with data. They should arrive with that knowledge. Engineers I think have a tendency to trivialize this, but true expertise in this domain comes only with years of experience.

* They should strive to find modeling solutions that are right for a particular business problem. If they seem to be only applying the hottest research regardless of the tradeoffs for the particular business problem, that's a red flag.

* Their focus should be on integrating themselves with the product/business as much as possible, and with the engineering team as much as possible. If they're expecting to be handed directives, that's a recipe for a ton of wasted time.

DS should never, ever be siloed into their own little DS world. They will be useless without a deeply intimate knowledge of the business goals, the needs of product, and the capabilities of the engineering team.

As they progress, they should become more and more "full-stack", otherwise they are stagnating.

Companies were in a rush hire "data scientists" and boot camps like Insight were more than happy to pump out very impressive PhDs with just enough understanding to build a Keras model.

I've worked in industry awhile doing DS work and have been astounded at the number of PhDs that both don't know how to write Python that doesn't live in a notebook and throw away years of disciplined experimentation experiences to just throw keras models at data until the needle moves.

There do exist excellent data scientists out there, who are both very solid software engineers and really know their stuff mathematically, but I've found most of these people can't reliably find jobs because the people interviewing them know so little that good data scientists will be penalized for answer a stock question correctly.

The field has been so flooded with amateurs that have no idea what they're doing, that potential mentors have been driven out, and now it's just a mess. To get a job doing DS if you do know what you are doing you have to play a weird game where you guess the incorrect answer the interviewer has in mind.

Sometimes I think a company (not having the DS experience themselves) mistakenly over-hire DS roles in today's hype of "AI" when their data is mostly run-of-the-mill and only requires simple linear models that can be architected an understood by a stats/math-savvy engineer. Even then, a good DS is still useful (even linear models can be complex: e.g. what priors do you want to use? Do you want a multi-task solution? etc.), but maybe not worth the cost.

There's a huge variety in DS responsibility and background between companies.

Maybe I'll come back in a few hours, but for now I'll stay away.

Reality is that data science is here to stay. It's coming out of the honeymoon period, and things may never be as hyped up as it has been the last decade, but that's probably a good thing for the field. Everyone will probably move on to hating the next up and coming thing. I have a hunch it could be something in data engineering because, while not exactly new, it is absolutely the next "data science" in terms of demand, and with products like Snowflake having so much hype behind them, it seems the backlash will be inevitable.

I remember an era before "data scientist" was a job title. When we (programmers) would analyze data to see if we had enough information available to identify the problem, if not, fix that, then come up with a strategy to solve it, test, and finally deploy the model. The fun part was trying different solutions and analyzing the data. It also felt awesome to deploy a product that worked like "magic." Product owners didn't know or care what a neural net was, they were just happy it worked.

Now there are tons of data scientist out there who take the easy, fun, rewarding work and try to skip over the nitty gritty implementation details. Then management thinks engineering is incapable of doing such work, and the only time we get the opportunity to do something fun is to do so behind the scenes.

From my experience, there are two types of data scientists who work who do infrastructure work: 1) Those who do not make the best data scientist because their skill set is too far in engineering land, leaving them weak where it counts. If the startup is relying on the data scientist to be profitable, I'd be cautious with these types. or 2) Someone who is senior, beyond senior really, who has worked both jobs, and doesn't mind doing both jobs. This unicorn is so rare it is mythical. The joke when the terminology was created is they're so rare no one has ever seen one, hence unicorn.

Me, I can not do the work I need to do if I'm on call. That is where I draw the line. That means hiring someone to monitor the infrastructure. Furthermore, I'm an okay architect, but you really do want to hire a specialist if you can help it for that. Do I help them with the infrastructure? Absolutely, but they're on call if a server is on fire. They have the admin login credentials, not me.

I get wearing multiple hats, but keep in mind to be a data scientist you're already wearing multiple hats. Being a data scientist is like double majoring and getting a phd. At what point are they stretched too thin? The consensus in the industry is they're already stretched too thin and should be broken up into different specialized roles.

>My question to HN is this; are there rolls at these larger companies for a Data Scientist who who primarily just crunches data in R and Python without the ability to actually build the pipelines / tools or conduct research?

That is the standard role, even at startups. However, the industry consensus these days is data scientists should have more responsibility when it comes to deploying models than previous standards.[1] So data scientists are being pushed in a more engineering direction, not with hosting sql servers and infrastructure, but with working with engineers to make sure the models are monitored properly. This change comes from model deployment being further automated as time goes on, making it easier for the data scientist to have more responsibility during this stage.

[1] source: https://www.dominodatalab.com/static/gfx/uploads/domino-mana... page 9. Suboptimal organization and incentive structures.

Having hired many data scientists/ML engineers over the years, people that build robust automated intelligence directly into products are extremely rare. I've estimated a maximum of 10K people in the entire world. They also command the highest salaries, not coincidentally. Very few people have both the statistics and engineering backgrounds as well as temperament to be successful, particularly when the problem requires new data sources or new types of models. There are some real simple practical hurdles such as the need to implement robust tracking that allows data snapshotting at the time when a decision needs to be made without affecting product performance, as well as figuring out how to gather data on users/situations that are actually important for moving the needle (first time users, casual users, etc.) There is also a mismatch between the best frameworks for prototyping/research and implementation which (at least at the companies I've worked for) can be summarized as "Java is good for application development, not ML. Python is good for ML, not application development."

You'd need someone with excellent communication skills (presentation, memo writing, teamwork), project management skills (identifying & overcoming workflow bottlenecks), professional skills (timely responses, political savvy), technical skills (application programming, advanced databases, advanced machine learning, Excel modeling) and finally some business domain knowledge.

This is an uncommon intersection of skills.

    DataEngineer = DataJanitor
      ∪ MlEngineer
      ∪ MlOps
      ∪ BusinessAnalyst

"Garbage in, garbage out" has been the mantra for a long time. Sure, there are tools and techniques to deal with not-so-good data, but those are add-ons, not a core part of the value proposition of ML.

I would love to be on that wait staff; as an infradev I feel like the process is very close to me but I am struggling to break into it.

But the statement is also a bit like saying you can use plumbers to design and build a chemical refinement plant which also just moves chemicals from point A to point B. Or you can design a citywide sewer system with a bunch of plumbers.

There are many cleansing, refining, orchestration, dependency, data quality, governance and optimization problems to be solved and a wide variety of tools that have for whatever reason never grown into higher-level open source frameworks and are thus reimplemented in various forms in many places.

Data engineering (somewhat like software engineering actually) doesn't require much if any of the math and physics I took in engineering courses in college, but it does require rigorous systems thinking about how to design and build structures that withstand adverse conditions that are thought patterns common to other engineering practices, so I don't think it's a totally crazy title for the role.

Then the federal government came up with this title something like "Secretary of Data Scientist" and the guy who made up the term worked at that.

In my experience, this is true:

> "data scientists" are expected to be the equivalent of full-stack engineers (or maybe more accurately: one-man CTO shops)—to understand data architecture, understand business architecture, ensure data quality, build data into product, build dashboards, derive insights, posit hypotheses, set strategy, and drive business value.

But this is not:

> Thus many "data scientists" are juiced-up report-builders who can't analyze their way out of a paper bag.

Rather, the data scientists are trained in only two of the requirements you mentioned: derive insights, posit hypotheses. The rest is all self-study and on-the-job experience. This means that we are putting unrealistic expectations on data scientist and/or their training is insufficient, not that data scientists are somehow morons.

None of this discussion is helped by the fact that companies want to get into data science without actually having much data strategy.