Hacker News Reader

Comment by amelius | Hacker News Reader

Before invoking a command, always first check if the length of the command is not longer than ARG_MAX. For example, if this is your command:

    grep --ignore-case --files-with-matches -- "hello" *.c

Then invoke it as follows:

    CMD="grep --ignore-case --files-with-matches -- \"hello\" *.c"
    ARG_MAX=$(getconf ARG_MAX)
    CMD_LEN=${#CMD}

    if (( CMD_LEN > ARG_MAX )); then
        echo "Error: Command length ($CMD_LEN) exceeds ARG_MAX ($ARG_MAX)." >&2
        exit 1
    fi

    eval "$CMD" # warning, evaluates filenames

View on HN

That might be sensible, but also obscure the script logic.

Since using Linux exclusively, I don't think I've ever encountered an issue due to too many arguments/length. And it's the first time I'm actively searching online for ARG_MAX.

I understand that different shells might be different, but with reasonable lengths is there any chance of it being relevant (aside from xargs, where it's generally intended, or better, to pass along each argument individually).

I started running into these issues when I started working with training examples in the context of deep learning. A folder with millions of files is then not unheard of.

Also if you do things like */*/*, then you can quickly get large command lines. Or even if you do long_name/another_long_name/*.

I've had problems with "cat *.csv" plenty of times processing data that is generated in many small files.

It is really difficult to deal with, because on top of the arg max limit, globs are not guaranteed to be in order.

The solution is not obvious and hard to get to if you don't know the foot guns in advanced and hard to read once implemented.

You should always type check your shell scripts as well. For example, you just:

    $ shelltypes script.sh
    # Welcome to shelltypes v 3.23.2
    # type ‘help’ if you’re stuck
    >>> {1) # import POSIX;;
    Importing 73 items.
    >>> {2} # append loadpath “/opt/local/shelltypes/base”;;
    >>> {3} # import base::YOURPROJECT;;
    Importing 15 items.
    >>> {4} # check “YOURSCRIPT.sh”
    Parsing YOURSCRIPT.sh.
    Reticulating splines.
    Expanding aliases.
    Analyzing free shell environment variables.
    Found inconsistencies in PATH.
    Warning: Low battery!!!
    Warning: found free type for ‘shred’, ignoring.
    Warning: use of sudo requires password under /etc/sudoers.
    Warning: this utility is fake.
    Error: use of cat impossible in the presence of mutt.
    Found 15 errors.
    Try again. Goodbye.
    $

Then you can be pretty sure your script isn’t going to do unnecessary harm, and has some proper guardrails in place.

Where does "shelltypes" come from? I can't find anything on DuckDuckGo or Google, but this seems like it would be very useful.

From the output in the post I'm going to assume it's either a joke post or an LLM hallucination.

Oh god, it even says right there:

> Warning: this utility is fake.

Well played! I guess I got too excited at the possibility of such a tool existing.

Such a tool does exist, it's called shellcheck. If you give /bin/sh shebang for example it will tell you off using non-POSIX features regardless of whether they'll work with the sh on your system.

(Personally I typically use bash though, largely so I can `set -eEuo pipefail` next.)

The thought is very compelling! shellcheck is pretty great, but of course isn’t this complete. It definitely can’t reticulate splines, for instance.

I don't know shelltypes, and it sounds like a linter for shell scripts.

Does shelltypes warn against a failure to check for ARG_MAX?

> eval "$CMD"

That means you will eval all the filenames, so if you have a file with spaces in it will appear as two files, if there is a `$` in the name it will trigger parameter substitution and so on for the other shell meta-characters.

Yes, that could be true. I'm not great in Bash. Be careful. These types of error are why I don't use Bash. I just wanted to give an example in a commonly used scripting language. The main point here is to check ARG_MAX.

If you are doing something where going over ARG_MAX is a real possibility it would be better to write your scripts to avoid the problem altogether rather than awkwardly try to detect it wit ha bonus exploit. For example, many commands can accept a list of files on standard input or from a list file.

On my system, `getconf ARG_MAX` is over 2m.

I have seen some heinously long cmdline strings, but nothing close to that. Usually when invocations have creeped up into the O(1k) character limit at places I've worked, I've implemented a "yaml as cmdline args" option to just pass a config file instead.

Have you seen scenarios where this is actually limiting?

Yes, if *.c expands to a string over 2m. Maybe that is a lot for .c files, but it may easily happen with .tiff and a folder full of images used for training a deep learning model, for example.

Thanks, this is interesting. I have done a lot of this sorta stuff (glob expanding giant directories) without a thought for this `ARG_MAX` libc parameter, but now I know I need to keep it in mind!

> Before invoking a command, always first check if the length of the command is not longer than ARG_MAX.

Tell that to Google and Mozilla. /s

Reply on news.ycombinator.com