Command-line Tools can be 235x Faster than your Hadoop Cluster (2014)

This is not true at all. Almost all the cloud providers have their own Hadoop distributions that is used a lot in many companies.

And when people say "won't fit in anything else", do explore your options before! RAM storage can be very, very, very big and the cost for using it is minuscule compared to what it used to be!

On that note, there is this great website for seeing if there are servers that can handle fitting all your data in RAM: https://yourdatafitsinram.net/

Systems like Power System E980 can handle up to 64TB RAM, which is a lot of data. Just like parent said here, do try to fit things on machines like this before even trying out large-scale storage clusters, because they are a hassle to deal with and generally not worth the cost.

Edit: For the Cloud-hosting crowd out there, I took a quick look at https://instances.vantage.sh/ and sorted it by memory. Largest instances you can get at AWS is "U-24TB1 Metal", which comes with 24TiB of RAM (and 448 vCPU so the computation itself gets as fast as possible too [granted you can parallelize it]), which should fit most cases of "big data" I've seen in the wild today. Unsure about running costs though, but I'm 90% sure it's cheaper to have that instance for a couple of hours than the cost for having to deal with storage clusters, which are also slower.

In general, this can be an effective approach, but at least fulltext search is another story.

Storing hundreds of MBs (actually, I think even tens of MBs can be problematic) in text files or a db like MySQL (whose FT engine is terrible) will result in slow fulltext searches.

For example I've got all my CD ripped to FLAC files, but my car only takes mp3 or wav... So I did a batch convert of FLAC to mp3, making sure to put all cores at work by piping the output of "find" into "parallel".

Some commands also allows to directly parallelize (like, say, "make -j 16 ..." to build using 16 cores/hyperthreads).

    function wait_all {
      for job in $(jobs -p); do wait "${job}"; done
    }

SQL is everywhere and it is fast and kinda portable. I have done sqlite analytics jobs over 4-6GB databases on a desktop machine 10 years ago to generate really complex reports. It worked wonderfully and was the shortest time from raw data (xml, html, csvs) to useful results.

This is so true. I write data pipelines at work. I only use SQL to move data around for this very reason. You never get any bug to fix with SQL. You sometimes need to adjust some queries to some new bad input but never ever will you see : 'index out of bound' or 'wrong argument' or whatever.

- be a clone/repo for disparate databases so you don't need to figure out access/security/location or impact production systems

- an interface to management types that aren't technical or don't have tech people to do these things

- should provide a "librarian" knowledge of the enterprise's data and data sources

- should have knowledge on how to analyze data using different tools

- be able to schedule movements/reports and manage that

If you don't need any of that, then ... yeah, don't use it. But those sets of requirements should be useful to anything that deems itself an "enterprise".

I'm sure there's some arbitrary lines that can be drawn, but anything under 1 TB is not big data anymore and can be processed on a single machine.

Other things to consider is data volume though. A popular use case of hadoop was to take e.g. server access logs - high volume data that traditionally you wouldn't hold onto for long - and get something meaningful out of that.

I think it's a bit more flexible than using cli tools since you can set some sort of time index and query specific timeslices fairly easily

InfluxDB does, indeed, have its own query language though.

I know this was a typo, but the idea of Oracle upselling someone on a quantum field theory based solution to a string matching problem amuses me greatly.

Whatever else keeps poping up against them are only usefull in special use cases, not needed for 90% of the common use cases, and even then, it isn't like relational database vendors are frozen in time without improving them.

The default advice is that you should catch a plane to go anywhere, and the author is showing why you should walk.

I thought parquet was columnar stored? Is this a fault of parquet or just the shear number of columns trying to get accessed?

I agree with your general premise though. I'd rather take a dirty dataset, throw it into S3, spin up a Redshift cluster, do what I need, spin down the cluster. You can work with billions of records fairly easily with plain old SQL and c-store databases.

A mere 7 machines is almost certainly going to be slower than 1.

There was a recent post here recently titled "Latency Numbers Every Programmer Should Know": https://news.ycombinator.com/item?id=30546995

In this case the dataset was small enough to fix in my laptop's DRAM without straining it. If we assume the 7 machines are in the data centre, that means the two numbers to compare are the main memory reference (100ns) versus the Data Centre round trip time (500us). That's a factor of 5,000.

If your intuition told you those 7 machines are going to be faster, then you really should invest the time to internalise those numbers. The article is 100% correct - every programmer should know them.

It's the same narrative people use for many complexity or slow things

Or is that more of a kubernetes thing?

Especially for those of us who get paid for developing, designing, supporting, architecturing, meetinging, catering, conferencing and maintaining everything around it :-)

Why not? Cron can schedule jobs, the documentation for standard shell tools is among the best, almost every developer can handle bash scripts, Logging can be done via syslog.