Hacker News Reader

Comment by sails | Hacker News Reader

It would be useful to give more clarity around your thoughts on the "Data Engineer" role. Is it also in decline? Is the market as a whole in a relative decline?

(I wrote a blog [1] making a call that Data Engineering was likely also to see something of a relative demand decline, or be better defined into Software Engineer and Analytics Engineer, so I am quite interested in your analysis here)

https://groupby1.substack.com/p/data-engineering

> Most businesses' data engineering needs have been solved or will shortly be solved by managed services that 10 years ago would require endless and extensive self-built ETL pipelines, databases and tools. For the exceeding majority of businesses, this means they can and should focus on building capacity for business logic, analysis and predictions instead of data engineering.

View on HN

> It would be useful to give more clarity around your thoughts on the "Data Engineer" role. Is it also in decline? Is the market as a whole in a relative decline?

My thoughts: the tech job market as a whole has been in decline, as unanimously observed. There may be some signs that the slowdown is abating though. Next 6-12 months will be key to see how DS and DE rebound (or not).

Could not disagree more with your take of "DE demand will decline due to DE needs being already solved for most businesses". Apologies, but have you ever worked as a data engineer or even close to one? Pipelines break, requirements change, businesses expand, and infrastructure needs to be managed and optimized, etc. ETL processes, in the wild, are decidedly not one-off affairs.

The evidence points to demand for data engineers declining on a relative basis to other data roles, following the data science trajectory. Agree or disagree?

Maybe the analysis as to _why_ is wrong, but that is what I'm trying to unpack.

HN Guidelines: "When disagreeing, please reply to the argument instead of calling names"

Anecdotally, DE demand has increased relative to DS as companies realize they need DEs and not DSes.

Currently demand for DEs outstrip DS

A lot of what modern data engineering has turned into is connecting various tools and software together. So adding another managed service doesn't feel like it is going to magically solve the problem. It is going to be just one more tool that the DE's will be managing. And indeed that has been my experience. For every tool that we added to the stack, we ended up spending just as much time fighting the tool as we did maintaining the self-built solution that the tool replaced. The two main advantages of using tools over DIY solutions is that they have an opinionated way of doing things, and they usually come with extensive documentation. So on boarding a new team member is easier. But engineering hours saved is pretty much a wash compared to DIY once you hit that first edge case that the tool does not handle elegantly.

> But engineering hours saved is pretty much a wash compared to DIY once you hit that first edge case that the tool does not handle elegantly

This is probably the exact crux of the argument, and I wonder if it reduces the overall demand relative to the growth in the industry. Do you need 10 engineers (like a decade ago) when 1 (great engineer) can deal with the bulk of the work with "better tools" and then DIY on the edge cases. This has been my experience with the modern ETL tools.

I don't have a good explanation as to why the "fraction of terms" for DE is going down. But the decline seems to be coincident with the general decline in tech hiring, so my best guess is that the DE role tends to be a bit more internal facing than other software engineering positions, so it is probably an easy target for hiring freezes. Based on my personal experience, the DE team seemed to always be the last team to get approved for additional hires, as the other teams in the engineering org where more directly tied to revenue generating initiatives whereas we were often seen more as a cost center.

But I don't think this is a long term trend. The DE role was originally tied heavily to the rise of data science, but it has turned into more of an operations role since then. You can probably get away with slowing down hiring for a little while, but just like with janitorial services, if you cut back too much things are going to get messy.

Reply on news.ycombinator.com