Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

yeah but who was getting better stuff on their resume? didn't you get the memo about perl?

Just because your throw-away 40 line script worked from cron for five years without issue doesn't mean that a seven node hadoop cluster didn't come with benefits. You got to write in a language called "pig"! so fun.

> over several days, to run a work load an order of magnitude smaller

Here I sit, running a query on a fancy cloud-based tool we pay nontrivial amounts of money for, which takes ~15 minutes.

If I download the data set to a Linux box I can do the query in 3 seconds with grep and awk.

Oh but that is not The Way. So here I sit waiting ~15 minutes every time I need to fine tune and test the query.

Also, of course the query now is written in the vendor's homegrown weird query language which is lacking a lot of functionality, so whenever I need to do some different transformation or pull apart data a bit differently, I get to file a feature request and wait a few month for it to be implemented. On the linux box I could just change my awk parameters a little bit (or throw perl in the pipeline for heavier lifting) and be done in a minute. But hey at least I can put the ticket in blocked state for a few months while waiting for the vendor.

Why are we doing this?

People should first try the simplest most obvious solution just to have a baseline before they jump into the fancy solutions.

I imagine your laptop had an SSD.

People who weren’t developing around this time can’t appreciate how game changing SSDs were then spinning rust.

I/O was no longer the bottleneck post SSD’s.

Even today, people way underestimate the power of NVME.

There is an incentive for people to go horizontal rather than permitting themselves to go vertical.

Makes sense, we are told that vertical has limits in university and we should prioritise horizontal; but I feel a little like the "mid-wit" meme, once we realise how vertical we can go then we can end up using significantly fewer resources in aggregate (as there is overhead in distributed systems of course).

I also think we are disincentivised from going vertical as most cloud providers prioritise splitting workloads, most people don't have 16TiB of RAM available to them, but they might have a credit card on file for a cloud provider/hyperscaler.

*EDIT*: Largest AWS Instance is, I think, the x2iedn.metal ith 128vCPU and 4TiB RAM

*EDIT2*: u-24tb1.metal seems larger; 448vCPU and 24TiB Memory, but I'm not sure if you can actually use it for anything that's not SAP HANA.

I think you have an off by one year error in these.

Hadoop has largely been replaced by Spark which eliminates a lot of the inefficiencies from Hadoop. HDFS is still reasonably popular, but in your use case, running locally would still be much better.

Most people who wasted $millions setting up Hadoop didn’t have data sets larger than could fit on a single machine.

This will not stop BigCorp to spend weeks to setup a big ass data analytics pipeline to process a few hundred MB from their „Data Lake“ via Spark.

And this isn’t even wrong, bc what they need is a long-term maintainable method that scales up IF needed (rarely), is documented and survives loss of institutional knowledge three layoffs down the line.

This is exactly the point of the article. From the conclusion:

> Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools.

What can be done on a single machine grows with time though. You can have terabytes of ram and petabytes of flash in a single machine now.

Even if you have a lot more than that you could easily use SQLite with FAISS. It works great.

You can easily add two zeros to that.

272MB/s is "consumer spinning rust RAID 5 via USB3" speeds. I know this because that's roughly what my NAS backup machine does on writes/reads. From a NAS on gbit it's overpowered by 150%, for sure.

Reminds me of one of my favourite twitter posts:

> Small Data is when is fit in RAM. Big Data is when is crash because is not fit in RAM.

https://twitter.com/DEVOPS_BORAT/status/299176203691098112

Actually the awk solution in the blog post doesn’t load the entire dataset into memory. It is not limited by RAM. Even if you make the input 100x larger, mawk will still be hundreds of times faster than Hadoop. An important lesson here is streaming. In our field, we often process >100GB data in <1GB memory this way.

Isn't a program really a scientific experiment? About as pure as you can get. It's not formalized as such but that's trivial.

There are two reasons creating software will never (in my lifetime) be considered an engineering discipline:

  1) There are (practically) no consequences for bad software.
  2) The rate of change is too high to introduce true software development standards.

Modern engineering best practice is "follow the standards". The standards were developed in blood -- people were either injured or killed, so the standard was developed to make sure it didn't happen again. In today's society, no software defects (except maybe aircraft and medical devices) are considered severe enough for anyone to call for the creation and enforcement of standards. Even Teslas full-self-driving themselves into parked fire trucks and killing the occupants doesn't seem enough.

Engineers that design buildings and bridges also have an advantage not available to computers: physics doesn't change, at least not at scales and rates that matter. When you have a stable foundation it is far easier to develop engineering standards on that foundation. Programmers have no such luxury. Computers have only been around for less than 100 years, and the rate of change is so high in terms of architecture and capabilities that we are constantly having to learn "new physics" every few years.

Even when we do standardize (e.g. x86 ISA) there is always something bubbling in research labs or locked behind NDAs that is ready to overthrow that standard and force a generation of programmers into obsolescence so quickly there is no opportunity to realistically convey a "software engineering culture" from one generation to the next.

I look forward to the day when the churn slows down enough that a true engineering culture can develop.

The way I see it is that we're in an era analogous to what came immediately after alchemy. We're all busy building up phlogiston like theories that will disprove themselves in a decade or two.

But this is better than where we just came from. Not that long ago, you would build software by getting a bunch of wizards together in a basement and hope they produce something that you can sell.

If things feel worse (I hope) that's because the rest of us muggles aren't as good as the wizards that came before us. But at least we're working in a somewhat tractable fashion.

The mathematical frameworks for construction were first laid out ~1500s (iirc). And people had been doing it since time immemorial. The mathematics for computation started about 1920-30s. And there's currently no mathematics for the comprehensibility of blocks of code. [Sure there's cyclomatic complexity and Weyuker's 9 properties, but I've got zero confidence in either of them. For example, neither of them account for variable names, so a program with well named variables is just as 'comprehensible' as a program with names composed of 500MB of random characters. Similarly, some studies indicate that CC has worse predictive power of the presence of defects than lines of code. And from what I've seen in Weyuker, they haven't shown that there's any reason to assume that their output is predictive of anything useful.]

"It's so funny how there is almost no 'science' - or 'engineering' - in modern 'computer science' or 'software engineering'"

It may not have been clear in 2014, but it is now: Data scientists are not computer scientists or software engineers. So tarring software engineers with data scientists practices is really a low blow. Not that we're perfect by any means, but that data point you're drawing a line through isn't even on the graph you're trying to draw.

I was unlucky enough to brush that world about a year ago. I am grateful I bounced off of it. It was surreal how much infrastructure data science has put into place just to deal with their mistake of choosing Python as their fundamental language. They're so excited about the frameworks being developed over years to do streaming of things that a "real" compiled language can either easily do on a single node, or could easily stream. They simply couldn't process the idea that I was not excited about porting all my code to their streaming platform because my code was already better than that platform. A constant battle with them assuming I just must not Get It and must just not understand how awesome their new platforms were, and me trying to explain how much of a downgrade it was for me.

"We don't have calculations for how to estimate, given X computing power, and Y system or algorithm, how much Z work it can do."

Yeah, we do, actually. I use this sort of stuff all the time. Anyone who works competently at scale does, it's a basic necessity for such things. Part of the mismatch I had with the data scientists was precisely that I had this information and not only did they not, they couldn't even process that it does exist and basically seemed to assume I must just be lying about my code's performance. It just won't take the form you expect. It's not textbooks. It can't be textbooks. But that's not the criterion of whether such data exists.

We do actually have some methods of calculating an expected performance. For instance we know that a Zen4 CPU can do 4 256 bit operations per clock, with some restrictions on what combinations are allowed. We are never going to hit 4 outright in real code, but 3.5 is a realistic target for well optimised code. We can use 1 instruction to detect newline characters within those 32 bytes, then a few more to find the exact location, then a couple to determine if the line is a result, and a few more to extract that result. Given a high density of newlines this will mean something on the order of 10 instructions per 32 B block searched. Multiply the numbers and we expect to process approximately 11 B per clock cycle. On a 5 GHz CPU that would mean we would expect to be done in 32 ms, give or take. And the data would of course need to be in memory already for this time to be feasible, as loading it from disk takes appreciably longer.

Of course you have to spend some effort to actually get code this fast, and that probably isn't worth it for the one-shot job. But jobs like compression, video codecs, cryptography and that newfangled AI stuff all have experts that write code in this manner, for generally good reasons, and they can all ballpark how a job like this can be solved in a close to optimal fashion.

It's all cycle. Remember when XML was the future ?

Money quote from https://www.bitecode.dev/p/hype-cycles:

> geeks think they are rational beings, while they are completely influenced by buzz, marketing, and their emotions. Even more so than the average person, because they believe they are less susceptible to it than normies, so they have a blind spot.

We need bigger big data (tm) to feed our Ai.

> but even that doesn't become as annoying and misunderstood as the big data fad was.

Must be nostalgia. AI is much, much worse. And, even more importantly, not only it is a annoying buzzword, it is already threatening lives (see the mushroom guide written by AI) and democracies (see the "singing Modi" and "New Hampshire Officials to Investigate A.I. Robocalls").

Also both OpenAI and Anthrophic argued if licenses were required to train LLMs on copyrighted content, today’s general-purpose AI tools simply could not exist.

This might be naive, but I agree that AI hype will never be as annoying as Big Data hype.

At least 90% when people mention wanting to use AI for something, I can at least see why they think AI will help them (even if I think it will be challenging in practice).

99% of the time when people talk about big data it is complete bullshit.

AI is arguably the most well known MapReduce that we have, today, though

for the given question, sure.. you can

there are 60tb ssd's out there.. you might even fit all of the 8tb on a given server

Unix has a split(1) tool.

"Can be 235x faster" != "will always be 235x faster", nor indeed "will always be faster" or "will always be possible".

The point is not that there are no valid uses for Hadoop, but that most people who think they have big data do not have big data. Whereas your use case sounds like it (for the time being) genuinely is big data, or at least at a size where it is a reasonable tradeoff and judgement call.

To people's beliefs on this, here's a Forbes article on Big Data [1] (yes, I know Forbes is now a glorified blog for people to pay for exposure). It uses as example a company with 2.2 million pages of text and diagrams. Unless those are far above average, they fit in RAM on a single server, or on a small RAID array of NVMe drives.

That's not Big Data.

I've indexed more than that as a side-project on a desktop-class machine with spinning rust.

The people who think that is big data are the audience of this, not people with actual big data.

[1] https://www.forbes.com/sites/forbestechcouncil/2023/05/24/th...

Have you read the article ?