This is great. It reminds of that Adam Drakes’ 2014 article “Command-line Tools can be 235x Faster than your Hadoop Cluster” [1] was the wake up call I needed.
1.
https://adamdrake.com/command-line-tools-can-be-235x-faster-...
We appeared to have a persistent bug though, processing left running overnight always seemed to crash. So after a couple of nights I stayed in the lab to try and see what the cause was. At 7:30pm the lab door opened and in walked the cleaner who reached down and unplugged the computer so that he could plug in the vacuum cleaner. Problem solved.
https://www.reddit.com/r/talesfromtechsupport/comments/5yrs1...
Let me know when your command line tools run multiple large scale processing jobs on petabyte datasets.
His article was him constructing a straw man about why people use Hadoop and attacking that.
How many people have used Hadoop for a project? How many Petabyte+ datasets do you think are out there?
Unless you truly believe the answer to those 2 questions are the same, I think you can see why that article had to be written.
I also appreciate quality of the writing and step by step journey.
When I first read the article, I found it timely as I felt bombarded by IaaS & PaaS offerings with “web scale” solutions for problems I couldn’t relate to, basic examples, and few case studies.