Hacker News Reader

Comment by hedora | Hacker News Reader

Each of the following is in the 1000 hours saved club (in no particular order):

jq, perl, grep, gnu textutils, gnu parallel, bash, xargs, gkrellm, nload.

Make deserves a special shout out. If you think it's only useful for building software, you've completely missed the point (similarly, if you think some language-specific tool is superior, you're doing things completely wrong):

Anyway, this document saved me at least 10,000 hours:

https://www.gnu.org/software/make/manual/html_node/index.htm...

Other peoples' python has the distinction of being the only software in my -1000 hours saved category. (And if you consider maintaining code written in the above "write only" languages, you'll realize how big of a feat that is!)

View on HN

> "If you think it's only useful for building software, you've completely missed the point"

I clicked on your link with 16 pages of just contents links to find what the point is, and found "the make utility automatically determines which pieces of a large program need to be recompiled, and issues commands to recompile them".

What's the point you see?

It automatically parallelizes any large scale data processing pipeline (assuming that your data set is already broken into files). People use it for things like documentation generation. I've used it to process the results of web scrapers, large scale data cleaning tasks, etc, etc.

It takes about an afternoon to set up NFS + SSH primitives that automatically distribute the computation across clusters of machines. Since it is restartable, it automatically tolerates hardware faults (up to dozens of machines, in practice).

Basically, you get Map Reduce, but for arbitrary data processing DAGs, and it supports any language that works well on Unix-style operating systems.

Okay but clearly none of this is "the point" of make, so your original comment is still baffling.

The point of make is that it's not only useful for building software. Take a look at the built-in implicit rules:

https://www.gnu.org/software/make/manual/html_node/Catalogue...

There are two things to note: (1) you can turn them off so they don't get in the way of other use cases, (2) they include non-build stuff like version control, linting, and document generation.

There's also a whole chapter on ar file maintenance. This is mostly useful for linking these days, but ar is the precursor to tar, so this used to also be useful for running filesystem backups:

https://www.gnu.org/software/make/manual/html_node/Archives....

Your point is that make has uses beyond that. That is not "the point of make"

No, that absolutely is the point of `make`. You quoted the "overview of make" page[0] which does say "The make utility automatically determines which pieces of a large program need to be recompiled, and issues commands to recompile them."

However, that same page, about 5 lines down, expands to say:

> Indeed, 'make' is not limited to programs. You can use it to describe any task where some files must be updated automatically from others whenever the others change.

Which is exactly what the original poster is claiming.

That's the beauty of open ended UNIX tools

The idea of working with a distributed data pipeline implemented via makefiles is nightmare fuel

$ make install

I think OP meant data analysis/one-offs as not building software that is technically true I guess

1000 hours each? Really? So literally half a year of full-time 40 hour weeks saved by each of these on your list, vs the next best alternative?

Yes. That statement was entirely literal; there is on reason to believe the statement "1000 hours" is anything but a precisely measured saving.

jq + parallel is a godsend. Take some command output, run another json-producing command in parallel on each line, and pipe it back into jq -s

Can you elaborate with a fake example?

> jq

Add xmlstarlet for anything that has to do with xml.

Reply on news.ycombinator.com