Disclaimer: not only did I upvote you but I totally agree with you. HOWEVER, it's not clear to me whether people are comprehending the tradeoff you describe. That is CRITICAL.
I keep hearing about Hadoop and Spark for graphs and models, but in actual practice (WHICH DEPENDS ON THE SIZE AND SHAPE OF THE DATA), an awful lot of data-feeding problems are more easily solved by columnar data stores, or graph data stores. For text mining and NLP, you are almost certainly better off with redundant unstructured storage due to the fundamentally unstructured nature of text and communications. For massively parallel model exploration sometimes it's better just to spin up a bunch of huge EC2 instances. Hadoop and/or Spark aren't necessarily critical aspects of solving the problem. Once you get a redundant, distributed infrastructure to run things on, you may not need to jerk around with name nodes and boring HA grunt work. Sometimes an existing purpose-built implementation (often on top of well-oiled infrastructures) does the trick. Sometimes not.
One interesting project I've kept an eye on (OK I lied, I'm a contributor) is storage of genomic data against a distributed, graph-structured reference. It's really fucking hard. Even with Google, UCSC, the Broad, and the usual heavyweights involved, the milestones have occurred on a timescale of years, and the eventual adoption is projected on a scale of decades. This is with some of the best in the world working on it, in every time zone. Again: DECADES.
So... I'm not against distributed computation. But you had better damned well know what you're getting into. I worked at Google. I work on GA4GH. If you can afford to take the long view, maybe your problems are complicated enough to invest in that sort of infrastructure.
But maybe they aren't, and Amazon or Google or Microsoft has already built what you need because they needed it too, and they hired a hundred graduates from the best engineering schools in the world to support their implementation. Is it worth reinventing the wheel when there are people racing at Formula 100 level out there? Only you can answer that.
There's a reason people don't swat flies with Buicks. Sometimes all you really need is a flyswatter.