Comment by chubot | Hacker News Reader

Comment by chubot | Hacker News Reader

Parent Full thread

OK thanks, looks like there are several features of GNU parallel that users like.

For the output interleaving issue, what I do is use the $0 Dispatch Pattern and write a shell function that redirects to a file:

    do_one() {
      task_with_stdout > $dir/$task_id.txt
    }

So if there are 10,000 tasks then I get 10,000 files, and I can check the progress with "ls", and I can also see what tasks failed and possibly restart them.

You even have some notion of progress by checking the file size with ls -l.

I tend to use a pattern where each task also outputs a metadata file: the exit status, along with the data from "time" (rusage, etc.)

But I admit that this is annoying to rewrite in every script that uses xargs! It does make sense to have this functionality in a tool.

But I think that tool should be a LANGUAGE like Oil, not a weirdo interface like GNU parallel :)

But thanks for the explanation (and thanks to everyone in this subthread) -- I learned a bunch and this is why I write blog posts :)

Thank you for writing this, it really crystalized for me why I feel the way I do about oil. I hate it. When I want a language, I want a real language like python not a weirdo jumped up shell (see what I did there?). What I want in a shell is a super small, fast, universally understood thing for basic tasks and easy expandability through tools like parallel and python.

For what it's worth, I consider oil to be closer to a unixy PowerShell rather than a more powerful bash. Note that this is not a slight, PowerShell is sweet for what it is. It (oil) really takes a hard left from the POSIX philosophy of focusing on one thing and doing it well. I'm also bitter that, if it's going to veer so far away from POSIX, that it didn't go the whole hundred and become a function language with comprehensions and such.

For what it's worth, everything you mentioned above about your approach can be done with parallel.

The point of oil that there are really basic things like safe quoting that shells should do well, yet none of the posix shell do!

Functional: there are interesting shells like Elvish. But it really goes PowerShell by adding internal rich data pipelines that dont have a unixy stream-of-bytes representation. Oil does NOT go that way; it works on stuff like QSN to make pure unix interconnects more robust.

Your `do_one`:

  * does not buffer stderr
  * does not check if the disk is full for a period of time during a task (thus risking incomplete output)
  * does not clean up, if killed
  * does not work correctly if task_with_stdout is a composed command

Given that GNU Parallel is a drop-in replacement for xargs, I am curious why you find it a 'weirdo interface'.

Any sufficiently NIH'd can be considered weird. -- Not Isaac Asimov

A lot of this comes down to familiarity. I tend to use "make -j 100" for what you're describing. If I write the Makefile carefully [1], it will handle resuming a half-finished job. I just looked and GNU parallel has a --resume argument which probably does something similar, and maybe with less hassle. But I don't do this often enough—and/or GNU parallel isn't "better enough"—that I'm likely to ever invest the time to learn GNU parallel.

btw, oil looks very cool. I hate how many footguns are in common shells.

[1] eg writing to a tempfile and atomically renaming into place: "task_with_stdout > $dir/task_id.txt.tmp && mv $dir/task_id.txt{.tmp,}"

Reply on news.ycombinator.com