This baseless assertion is patently wrong on so many levels. Building computing clusters on COTS hardware is a very mundane problem. Running processing jobs on data shards is a very mundane problem. Scaling COTS clusters transparently is a very mundane problem.
Many people use/used Hadoop for problems that did not warrant the overhead and complexity that comes with Hadoop. I've seen it countless times with my own eyes that people pre-emptively use tools like Hadoop and Spark because of a chance that they will hit a massive scale in the future.
This happens in both startups and enterprises alike: people like to think they have big problems too often.
A.k.a. resume-driven development. Having Hadoop on your CV looks sexier than awk.
I just sat there thinking I could probably run what they did on my phone.
And just to be clear - it wasn't a PoC or a demo.
Worse still, it didn’t even use HDFS and we eventually got sick of the crappy embedded Zookeeper/Kafka setup.
No, it still remains astonishingly wrong. Even container orchestration platforms are being adapted to provide the same service that Hadoop has been providing for years, and no one in their right mind would claim that running processing jobs on the cloud is a problem that almost no one has.
- Fits on one computer (most of the market)
- Fits on several computers (most of the rest)
- Requires a significant cluster of machines (50+ to store it)
Hadoop only really solves the last one. It has huge overheads in terms of speed and in terms of resources and headcount to run it properly, so it only makes sense at a particular scale. It's like a mainframe – most companies shouldn't buy one.
If you add to this the fact that Hadoop was about batch processing, and its "realtime" capabilities were poor, there really aren't that many potential customers, and many of the potential customers would rather run it in-house, or build their own system.
There's one category above that, which is "Fits in memory" and that is a huge chunk of the market. I've seen first hand people getting way too cute and complicated planning for scale, and then it works out that they don't even have more than a couple of GB of data.
Unless you're storing media, or you are truly "web-scale", your business data will very likely fit in 512GB.
I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets.
I've heard of companies doing extensive work to set up Hadoop to process a few GB.
Roughly I'd suggest that "fits on my laptop" is <1TB, "fits in memory" is < 1TB, "fits on one computer" is < 10TB, "fits on a small cluster" is <100TB, and "might be worth Hadoop" is >100TB. I could be too low on these though.
From Heroku's site, Heroku's 4GB database plan goes for 50$/(instance.month) while Heroku's 8GB plan goes for 200$/(instance.month).
Therefore, it isn't a question of if it makes finantial sense (it does) but how long the startup plans to operate to recover their investment.
> I've heard of a company refusing to purchase an external drive for an employee so they could process a handful of ~50GB datasets on their MacBook Air – instead forcing them to use "the cloud" or constantly download and backup datasets.
I find it rather strange how someone believes that it's a decent idea to conduct a company's data analysis work on what an employee manages to fit on an external HD, as it creates a whole lot of hard problems both legal and technical. I mean, how do you ensure the data's provenance is tracked and other data analysts can access the data? Who in their right mind would put himself on a situation where a minor lapse or misfortune (losing/getting the HD stolen) could put the company at risk?
> Roughly I'd suggest that "fits on my laptop" is <1TB, "fits in memory" is < 1TB, "fits on one computer" is < 10TB, "fits on a small cluster" is <100TB, and "might be worth Hadoop" is >100TB. I could be too low on these though.
That's a rather naive and missinformed take on Hadoop. Hadoop might be conflated with big data but it's actually a distributed system designed to reliably process data shards without having to incur a penalty to move data around. It makes absolutely no sense to base your assertion on data volumes alone. What matters if it the performance increase justifies setting up a hadoop cluster with the resourses available to a company.