rainboOow9

It is still ongoing, but the trend is really on managed services. Most shops that are still running hadoop distribution are doing it for legacy reasons (and I used to work in one). I mean, just look at job offers: how many offers do you see where hadoop experience is a plus VS cloud experience?

I am only speaking for myself here, but I am really feeling the switch from "data engineering" to "data ops", for whatever that means.

In short, 5-10 years ago, writing mapreduce / spark jobs (or even debugging / optimizing hive jobs) was complex enough that it was often the job of the data engineer (and not the data analyst / scientist). And I do not only mean writing the data processing logic, but more importantly, properly configuring it so that the resource footprint was acceptable. This required a good understanding of the underlying framework, analyzing the job execution plan, tweaking the resource configuration, etc.

Now, writing distributed jobs is pretty trivial with most cloud providers, hence it is now purely done by data analysts and scientists. And the data engineers have switched to doing more of a devops kind of work, doing the plumbing between the various cloud components and the IaC required to provide those cloud resources to other data users. In short, you can be a data engineer and have absolutely no clue on how distributed systems are actually working, this will not be an issue in your daily job.

This is spot on, and personally, the main reason why I decided to move to an engineering manager after 10 years as a developer.

And even as an engineering manager, I do not feel safe. I think only once you reach director level, you are protected from market hype and newest frameworks trends.

Ironic thing here is that Hadoop is mostly already outdated.

Which is btw one of the depressing thing for a lot of data engineers: we used to play with those cool distributed processing frameworks, and now? We are mostly writing some terraform to deploy cloud resources, most of the distributed part being handled by those cloud providers.

But is the system really responsible for all of this? I mean, if we go back to a more "natural" state, with simpler jobs and with more direct impacts, are we actually happier? Is cutting woods all day long more fulfilling that a tech job? Or being a mason and putting block on top of each other for other people that you do not know about?

Or is it just that in tech, we have the luxury to think about all of that, and to potentially take several years without working much thanks to our high salary?

Do we have some tangible numbers on the long term situation of those "quitting tech" people? Like, ok, they open a wind-surfing school, but then what? Does that work, are they more fulfilled and able to work the next 10+ years in that situation without burning out? Or do they go back to tech after a while because grass is always greener, and a job is a job and there are not as much fulfilling jobs as we would like to be.

Looking around me, I do know a fair amount of people quitting tech. To become a sport or management coach, a sound designer, a bar owner, a writer, etc. But a lot of those people seem to struggle even more now than they were when being in tech.

Is it though? Seems pretty specific to the tech industry (and some others).

Never heard that there were crazy interview rounds for experienced lawyers, marketers, project managers, even other non-tech engineers. After a certain amount of experience, people assume from your resume that you are not an impostor and interviews are mostly about motivations / behavourial fit.

What baffles me though is that its not like this crazy interview gauntlet is being maintained purely by HR. Engineers in those big tech companies are very protective towards those processes as well, and loudly protest if one would try to change them (complaining about lowering the bar, not hiring top tier talent anymore, etc.). And tbh, I would be afraid of what they would replace those LC interviews with. At least, I can prepare and grind a LC test. I do not want to have to explain in details what a inode is when interviewing for a back-end position, while I have never had a need for that kind of knowledge in 10 YoE.