Hacker News Reader

Comment by kylek | Hacker News Reader

Maybe not any better syntax, but xargs is usually installed by default in most distros, and I've never found parallel to be advantageous over it. I don't think this particular wheel needs to be invented again.

View on HN

xargs does nothing to prevent interleaving of output from simultaneously executed commands.

Yes. I use parallel if I need coherent output; otherwise I use xargs (which is significantly faster).

I suspect that's intended behavior. But if I'm understanding correctly, I think you could account for it with something like-

... | xargs -P10 -i sh -c 'command {} 1>{}.out 2>{}.err'

or maybe with tee:

... | xargs -P10 -i sh -c 'command {} | tee -a {}.out'

It would require your input not to have any special shell chars - e.g. no space.

It is definitely not something I would dare to run in production.

> xargs is usually installed by default

xargs is part of POSIX, so it should be everywhere.

(-P/-n are not, but both GNU and BSD versions do support it.)

xargs is great but doesn’t run the commands in parallel or chunk input does it?

It can, check the -n and -P arguments.

-P sets parallelism, -n number of arguments for each execution. Those are my main weapons. (-i is usually also necessary, and can allow for some pretty creative situations)

I don't swear on HN, but Holy Shit, I just added -P to xargs in a testsuite bash script, and it's 7 times faster.

I only have quad core, but -P values higher than 4 give better performance. Maybe because the tests take different amounts of time, and it avoids them backing up. To get x7, I had -P equal to the number of tests.

If the tests are ever waiting for I/O or something, or otherwise using less than one full CPU core, then I think that could explain why running more than the number of cores would be an improvement. If you run "top", or use the "time" utility, what fraction of a CPU core does each test use?

Good thinking, but they don't do any file io, and I'm not sure how it could use less than a full CPU (wouldn't it just finish sooner? Though waiting for io could do that). OTOH it's java (on android, so dex format) and does some classloading of user classes... so maybe that is io? Here's time results:

  real    0m1.818s
  user    0m2.020s
  sys     0m0.900s

top. (It completes in 1 sec, so put in a while loop - BTW top isn't behaving normally, "1" not giving multiple CPUs and not fitting on screen). Jus the top line:

  User 51%, System 23%, IOW 0%, IRQ 0%

xargs can do both and it rocks. https://www.google.com/search?q=xargs+parallel

It does. What xargs can't do is run stuff in parallel across many machines.

Yeah, you'd have to combine it with something like

https://github.com/erikfrey/bashreduce

Or pssh

I dislike parallel’s confusing options, but xargs does not report on progress.

Reply on news.ycombinator.com