Hi, working for major telco here.

I think the realisation is that most data workloads aren't really big data, which people already knew back in 2015, but the same people will overengineer things just to learn stuff. The fad moved onto something else, I agree.

But my department is a giant, just-ETL, everything-requires-a-cluster kind of department. More than half of our business runs on Hadoop, for a reason.

The Hadoop ecosystem is a bit like K8s. If it feels too complicated for your purpose, it probably is. But with all its warts, when your data is big enough, either you use the Hadoop ecosystem or you reinvent significant parts of the wheel.