Hacker News Reader

I've seen Big Data pipelines (Hive tables, Spark jobs for queries, data engineers setting it all up) for what was ultimately a 5-10 GB dataset.

Companies cargo culting Big Data stacks when their data is nowhere near big is very prevalent.

This a structural problem not an awareness one. Is not like they don’t know they can run it on a 5$ VPS, the problem is that there are no incentives to do so. You’d be surprised to know how much of engineering is there to address organisational challenges rather than technical ones (ie: micro-services)

Same. They've been sold a bill of goods by big cloud tech and company.