>I wonder how much of this kind of stuff exists out of necessity and how much of it exists because very smart people are just bored and/or unsatisfied.
That's a ton of it. Like it or not, publishing a digital newspaper is not a hard or unsolved problem; it's one of the web's core competencies. If you hire people who want to build cool stuff to supervise a CMS, well, you get this kind of outcome.
The raw cost is understated because these experimental setups misinterpret the functionality of the new architectures/formats they're using. It doesn't truly rear its ugly head until there is a major data loss or corruption event. It's not that these never happen with RDBMS, it's just that RDBMS contemplates this possibility and tries to make it pretty hard to do that, whereas message queues just automatically delete stuff (by design, so they can serve as functional message queues!).
RDBMS have spoiled us and we take its featureset, 40+ years in the making, for granted. We need to be careful and not assume that `GROUP BY` is the only thing we leave on the table when we "adopt" (more accurately abuse) one of these new-wave solutions as a system of record.
Since no one is going to admit to their boss "this wouldn't have happened if we used Postgres", and since most bosses are not going to know what that means, most of these spectacular failures will never be accurately attributed to their true cause: developers putting their interest in trying new things above their duty to ensure their employer's systems are reliable, stable, and resilient.