Hacker News Reader

Comment by speedplane | Hacker News Reader

> Maybe you see TB level processing a lot in your line of work, but most developers never will.

I doubt this is the case. If you're working on even a medium sized team, you'll see this level of data, whether it's internal server logs, publicly-sourced data, or a variety of other applications. Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run it on one machine). I don't know the percentage of developers that deal with this, but I do know that it's pretty common.

View on HN

> Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run it on one machine).

This doesn't follow:

1) One reason to run a cluster is to use flaky commodity hardware instead of high-reliability specialized hardware. Note the transition from specialized hardware and computer rooms of the '80s and '90s to cloud computing on preemptible instances.

2) They might just be straight-up wrong / misguided. http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... As a professional developer I've seen tons of distributed systems that could be replaced by a well-designed non-distributed system, but ELK and Mongo and etcd and Kafka and more generally cloud servers are easy off-the-shelf tools.

Just a few weeks ago I was working with an 11TB dataset which I processed with just unix command line tools told on a single Linux VM.

You can get a lot of milage out of parallel xargs and other of the shelf tools.

Indeed! Gnu prallel, xargs, make -j, netcat, mawk/gawk (with C extensions if needed), jq, coreutils, textutils can get insane amount of stuff done.

I have a simple bash script composed of 6 sed commands piped together to convert >100GB csvs (table dumps of recommendation data) into Redis binary protocol which are then ingested into a Redis Cluster using redis-cli --pipe. It takes somewhere around 15 minutes running on a modest bare metal server.

Sampling or subetting your data is almost always the correct response, especially for analysis.

If you're literally processing TB or PB of data, you'll want to parallelise. Though shell tools do this amazingly well.

Reply on news.ycombinator.com