Command-line tools can be faster than your Hadoop cluster

It was basically a few compiled perl scripts and some obfuscated shell scripts with a layer of glitz. People actually used it and LOVED it... It was supposedly better than the real tools some groups were using.

It was one of the more epic work trolls I've ever seen!

Also, hadoop is so painfully slow to develop in it's practically a full employment act for software engineers. I imagine it's similar to early ejb coding.

I think the point the author is making is that although they knew from the start that Hadoop wasn't necessary for the job, many people probably don't.

Most of us don't have scaling issues or big data, but that sort of excludes us from using all the fancy new tools that we want to play with. I'm still convinced that most of the stuff I work on at work could be run on SQLite, with designed a bit more careful.

The truth is that most of us will never do anything that couldn't be solved with 10 year old technology. And honestly we should happy, there's a certain comfort in being able to use simple and generally understood tools.

Some of my wording is a bit terse; sorry! :-) The article is great and I really enjoyed it. He's certainly got the right solution for the particular task at hand (which I think is his entire point) but he's generally right for the wrong reasons so I pick a few holes in that: I'm not trying to be mean. :-)

----- Classic system sizing problem!

1.75GiB will fit in the page cache on any machine with >2GiB RAM.

One of the big problems is that people really don't know what to expect so they don't realise that their performance is orders of magnitude lower than it "should" be.

Part of this is because the numbers involved are really large: 1,879,048,192 bytes (1.7GiB) is an incomprehensibly large number. 2,600,000,000 times per second (2.6GHz) is an incomprehensibly large number of things that can be done in the blink of an eye.

...But if you divide them using simple units analysis; things per second divided by things gives you seconds: 1.383. That's assuming that you can process 1 byte per clock cycle which might be reasonable if the data is small and fits in cache. If we're going to be doing things in streaming mode then we'll be limited by memory bandwidth, not clock speed.

http://www.techspot.com/review/266-value-cpu-roundup/page5.h... reckons that the Intel Core 2 Duo E7500 @ 2.93GHz) has 7,634MiB/s of memory bandwidth for reads.

That's 8,004,829,184 bytes per second.

Which means we should be able to squeeze our data through the processor in...

bytes per second divided by bytes = seconds =>

>>> 8004829184 / 1879048192.0 4.260044642857143

so less than 5 seconds.

We probably want to assume that there are other stalls and overheads, but a number between 20 and 60 seconds seems reasonable for that workload (he gets 12): the article says it's just a summary plus aggregate workload so we don't really need to allocate much in the way of arithmetic power.

As with most things in x86, memory bandwidth is usually the bottleneck. If you're not getting numbers with an order of magnitude or so of memory bandwidth then either you have a arithmetic workload (and you know it) or you have a crap tool.

Due to the memory fetch patterns and latencies on x86, it's often possible to reorder your data access to get a nominal arithmetic workload close to the memory bandwidth expected speed.

His analysis about the parallelisation of the workload due to shell commands is incorrect. The speedup comes from accessing the stuff straight from the page cache.

His analysis about loading the data into memory on Hadoop is incorrect. The slowdown in Hadoop probably comes from memory copying, allocation and GC involved in transforming the raw data from the page cache into object in the language that Hadoop is written in and then throwing them away again. That's just a guess because you want memory to fill up (to about 1.75GiB) so that you don't have to go to disk. That memory is held by the OS rather than the userland apps tho'.

His conclusion about how `sleep 3 | echo "Hello"` is done is incorrect. They're "done at the same time" because sleep closes stdout immediately rather than at the end of the three seconds. With a tool like uniq or sort it has to ingest all the data before it can begin because that's the nature of the algorithm. A tool like cat will give you line-by-line flow because it can but the pipeline is strictly serial in nature and (as with uniq or sort), might stall in certain places.

He claims that the processing is "non-IO-bound" but also encourages the user to clear the page cache. Clearing the page cache forces the workload to be IO bound by definition. The page cache is there to "hide" the IO bottlenecks where possible. If you're doing a few different calculations using a few different pipelines then you want the page cache to remain full as it will mean that the data doesn't have to be reread from disk for each pipeline invocation.

For example, when I ingest photos from my CF card, I use "cp" to get the data from the card to my local disk. The card is 8GiB. I have 16GiB of RAM. That cp usually reads ahead almost the whole card and then bottlenecks on the write part of getting it onto disk. That data then sits in RAM for as long as it can (until the memory is needed by something else) which is good because after the "cp" I invoke md5sum to calculate some checksums of all the files. This is CPU bound and runs way faster than it would if it was IO bound due to having to reread all that data from disk. (My arrangement is still suboptimal but this gives an example of how I can get advantages from the architecture without having to do early optimisations in my app: my ingest script is "fast enough" and I can just about afford to do the md5sum later because I can be almost certain it's going to use the exact same data that was read from the card rather than the copied data that is reread from disk and, theoretically, might read differently.)

He's firmly in the realm of "small data" by 4 or 5 base 10 orders of magnitude (at least) so he's nowhere close to getting a "scaling curve" that will tell him where best to optimise for the general case. When he starts getting to workloads 2 or 3 orders of magnitude bigger than what he's doing he might find that there are a certain class of optimisations that present themselves but that probably won't be the "big data" general case.

Having said that, this makes his approach entirely appropriate for the particular task at hand (which I think it his entire point).

Through his use of xargs he implies (but does not directly acknowledge) that he realises this is a so-called "embarrassingly parallel" problem. -----

I wouldn't typically reach for a big data solution short of hundreds of gigs of data (which is borderline, but will only grow from there). I might even reach for something like ElasticSearch as an interim step, which will usually be enough.

If you can dedicate a VM in a cloud service to a single one-off task, that's probably a better option than creating a Hadoop cluster for most work loads.

If you're only working one app on less than 1 TB, maybe you don't need something as complex as Hadoop. But given that a cluster is easy to setup (I made a really simple NameNode + Two Data nodes in 45 minutes, going cold), it might not be a bad idea.

I'll take this further and say that some tools for Hadoop that are not from Apache are really nice to work with even in a for non-Hadoop work. For example, I've got to join several 1 GB files together to go from a relational, CSV model into a Document store model. Can I do this with command line tools? Maybe. Cascading makes this really easy. Each file family is a tap. I get tuple joins naturally. I wrote an ArangoDB tap to auto load into ArangoDB. It was fun, testable and easy. All of this runs sans-hadoop on my little MBP.

Fun fact about the Cascading tool set is that I can take my little app from my desktop and plop it onto a Hadoop cluster with little change (taps from local to hadoop). Will I do that in my present example? No. Can I think of places where that's really useful? Yes, daily 35 fraud models' regression tests executed with each build. That's somewhere around 500 full model executions over limited, but meaningful data. All easily done courtesy of a framework that targets Hadoop.

GUN parallel should be a widely adopted choice. Lightweight. Fast. Low cost. Extendable.

Just curious

Here's what the "sensible adults" think about when they see problems like this. Operational Supportability: How do you monitor the operation ? Restart Recovery: Do you have the ability to restart the operation mid way through if something fails ? Maintainability: Can we run the same application on our desktop as on our production servers ? Extensibility: Can we extend the platform easily to do X, Y, Z after the crawling ?

I can't stand developers who come up with the xargs/wget approach, hack something together and then walk away from it. I've seen it far too often and it's great for the short term. Dreadful for the long term.

zcat/cut/sed/grep/awk/perl crawled through it in a couple of minutes and required less than half an hour to craft a reliable enough implementation (including looking up relations from foreign keys straight from the SQL dumps).

My colleagues, who still don't get the point of a command line, would still be restoring each dump individually and running SQL requests manually to this day (or more probably declare it "too complex" and mark it as a loss). Side note: I'm torn between leaving this place where nobody seems to understand the point of anything remotely like engineering or keeping this job where I'm obviously being extremely useful to our customers.

If so, using wget is a poor solution. I have not used wget in over a decade but as I recall it does not do HTTP pipelining; I could be wrong on that - please correct me.

I do recall with certainty that when wget was first written and disseminated in the 1990's, "webmasters" wanted to ban it. httpd's were not as resilient then as they are today, nor was bandwidth and hardware as inexpensive.

HTTP pipelining is a smarter alternative than burdening the remote host with thousands of consecutive or simultaneous connections.

Depending on the remote host's httpd settings, HTTP pipelining usually lets you make 100 or maybe more requests using a single connection. It can be acomplished with only a simple tcpclient like the original nc and the shell.

In any event, the line about a "distributed crawler" is spot on. Never understimate the power of marketing to suspend common sense.

Also, I find that I can often speed my scripts up a little by using exec in shell pipelines, e.g., util1 |exec util2 or exec util1 |exec util2.

There are other, better approaches besides using the builtin exec, but I will leave those for another day.

1: http://doc.scrapy.org/en/0.24/intro/tutorial.html

I built ag for searching code. It can be (ab)used for other stuff, but the defaults are optimized for a developer searching a codebase. Also, when writing ag, I don't go out of my way to make sure behavior is correct on all platforms in all corner cases. Grep, on the other hand, has been around for decades. It probably handles cases I've never even thought of.

HDFS is a psudeo block interface. If you have a real filesystem like lustre, or GPFS, not only do you have the abilty to use other tools, you can use that storage for other things.

In the case of GPFS, you have configurable redundancy. Sadly with lustre, you need decent hardware, otherwise you're going to loose data.

In all these things, paying bottom dollar for hardware, forgoing support is a false economy. At scales of 1pb+ (which is about 1/2 a rack now) its much much cheaper to use off the shelf parts with 24/7 support than "softwareing" your way out.

Because controversial topics are always fun! d:-)

The advantage of using cat, therefore, is the few seconds of laziness saved in not reading the manual.

Also, the cat | grep pipeline is illustrative. I remove it at the end.

Edit: I see they did use awk later in article, I should really read all of things before commenting.

I always throw this analogy to people who misunderstood Hadoop: A stone to crack an egg or a spoon?

Hadoop and RDBMS only have a thin overlapping region in the Venn diagram that describes their capabilities and use cases.

Ultimately, it is cost vs efficiency. Hadoop can solve all data problems. Likewise for RDBMS. This is an engineering tradeoff that people have to make.

Why would you want to use a database for this problem? The input data would take time to load into an ACID db and we're only interested in a single ternary value within that data. The output data is just a few lists of boolean values so it has no reason to be in a database either.

This is a textbook stream processing problem. Adding a database creates more complexity for literally no benefit assuming the requirements in the linked article were complete. I would be baffled to see a solution to this problem that was anything more than a stream processor, to say nothing of a database being involved.

1. A pipe to feed the file names into xargs for starting up parallel `mawk` processes.

2. A pipe to a final `mawk` process which aggregates the data from the parallel processes.

There's still some performance that could be gained by using a single processes with threads and shared memory, but this is pretty good for something that can be whipped together quickly.

There are command line tools there, but in my 10 years of being a Windows developer, GUI tools were more the norm.

There's a time and a place for both. Now developing predominantly under Linux, it amazes me how time consuming and clunky some tasks are on the command line compared to using a GUI (e.g. debugging, Visual Studio is just a fantastic IDE), but also how much faster and easier other tasks are with a CLI.

I still use programming languages like AWK to this day (I even list it as a known language on my resume; someone commented on it once, it's always a +1 for the company if they do). Recently though I've been exposed to some stuff in the Windows world that makes me think that, sure, they evolved slower but they evolved in a really interesting direction and possibly pulled themselves out of the local maxima that is the Unix world. (I'm referring to PowerShell.)

It's a little weird to me when devs don't immediately go to the command line, given that's just how I learned, but you've gotta recognize that it's not a flaw that they don't. Everyone just learned a little differently.

This is not even close to true. Of the developers that I know, many of them quite serious, live in the Windows world, and are very happy with for example VS or eclipse. This is very likely to be the case in most BFEs.

We get it. Your data's pretty big.

I very much work in "big data" with about 2 terabytes of new data coming in every day that has to be ingested and processed with hundreds of jobs running against them. The data needs to be queryable via an SQL like language and analyzed by a dozen data scientists using R or Map Reduce.

There isn't anything on the market today that has been proven to work in environments like this and has the tooling to back it up. Unless you want to prove everyone e.g. Netflix, Linkedin, Spotify, Apple, Microsoft wrong ?

1. Earth is 40 megameters in circumference. 40Mm / 50M = 0.8m