Hacker News Reader

Comment by theamk | Hacker News Reader

The only difficult part here is xargs parallel invocation. If you do this often, you may want to find/make the library for it. But even with just default python install, it is pretty straightforward:

     files = [f for f in glob("*.jpg") if "2019" in f]
     commands = ["convert ... {} album/{}".format(f) for f in files]
     subprocess.run("xargs -L 1 -P 32".split(), input=os.fsencode("\n".join(commands)))

Note that this code, as well as original shell version, breaks if the filenames contains spaces or quotes; this is trivial to fix in python, but hard to fix in bash.

View on HN

In sh, the correct version would be

    for f in ./*2019*.jpg ; do
        convert "$f" album/"$f"  # just don't use xargs
    done

Note the prepended ./ which is necessary if you want to protect against filenames that look like command line switches (a file named "-rf" can be pretty dangerous)

If you wanted to use xargs and still make it perfectly safe with regards to spaces and quoting, you could use a GNU extension:

    find . -maxdepth 1 -name '*2019*.jpg' -print0 | xargs -0 convertalbum.sh

    # convertalbum.sh
    convert "$1" album/"$1"

find prints the files terminated by ASCII NUL characters, xargs can understand this format if you give it the -0 option.

There are still problems around detecting errors in pipes, though.

> just don't use xargs

the whole point of my example was to show how easy parallelism is. You create a list of tasks, then you pipe the list to xargs or gnu parallel.

You can't write ASCII NUL characters from portable shell, that's why this task cannot be done correctly (for all possible filenames) from shell. You need to use tools like find(1)

So, no, it's not easy, at least not if you want to handle all cases without wreaking havoc.

Practically speaking, you can write NUL from shell because bash is everywhere, busybox ash emulates a lot of bash, etc.

However I won't claim it's pretty or easy for people to use, so the point of Oil is basically to clean up patterns like this, e.g. find -print0 | xargs -0 and more.

This post uses a related problem and solution as design motivation for Oil:

How to Quickly and Correctly Generate a Git Log in HTML

http://www.oilshell.org/blog/2017/09/19.html

> Note that this code, as well as original shell version, breaks if the filenames contains spaces or quotes; this is trivial to fix in python, but hard to fix in bash.

Your code sort of cheats, since it is calling xargs to do the parallelism. Also, it is missing the imports. The equivalent pure python version will be probably still more complicated (regardless of it working for general filenames or not).

> Your code sort of cheats, since it is calling xargs to do the parallelism.

This is the whole point though -- you can still invoke external binaries in python!

For example, if you need to sort 100GB file, direct Python approach will likely OOM... while /usr/bin/sort will work just fine. I have seen people use this as an excuse ("I cannot use python's sort(), so I will just rewrite everything in shell") -- but for a complex script, it is almost always better to use Python as much as possible.

Reply on news.ycombinator.com