Hacker News Reader

Comment by amelius | Hacker News Reader

I still get "Argument list too long" on a machine with 64GB of memory.

This is not acceptable in this day and age.

Most command line programs are open source so you can do something about that if it really makes sense.

You can't fix this in the tool you are trying to invoke because the tool never runs. The error E2BIG is returned from the syscall `execve` because it fails to run the program.

If you just want to work around this on some architectures you can simply raise the stack limit which defaults to 8MiB.

If you want to do this in a portable way, learn to use `xargs` or `find`.

> If you want to do this in a portable way, learn to use `xargs` or `find`.

xargs and find are a hack here, because it will start the target program multiple times. They only work if CMD FILE1 FILE2 has the same meaning as CMD FILE1; CMD FILE2.

The fix with the stack limit is also a hack. What other system requires you to specify a hard limit, these days?

-N 1 will invoke the command with only one arg at a time

It might protect you from shooting yourself in the foot.

It might also do the opposite.

Pretty sure that limit doesn't exist anymore.

I think it does. The man page for execve(2) documents it. On my system, the documented limit is 2 MiB.

OTOH, I do sort of see the argument for "why is there an arbitrary limit here?", beyond just "it's limited by available memory".

OTOH, what on Earth is one doing, passing more than 2 MiB of arguments to a program? Surely one would be better served with piped input, at that point…

I've hit command line length limits on both Windows and Linux (and they're different lengths).

Typically when I've run into it, it's been with build systems, especially those built around make/gnumake. Have a set of object files that need linking, what do you do? Typically, just expanding the variable holding the list as an argument list to ld or whatever works just fine. Until the project grows substantially and you're linking hundreds or thousands of object files. And what if they're in a different path? A subfolder? What if you're using absolute paths to object file locations? These things can blow up in weird and surprising ways and it's not always obvious what the cause is.

Thankfully, most tools used to build software typically have a mechanism to read arguments from a file that doesn't suffer from this limitation.

Also, as a sibling comment mentions: globs expansions. Typically done by the shell, then passed to the invocation target. Better to pass the glob in quoted and have invoked program do the expansion...

Just a couple places I've seen the problem.

> OTOH, what on Earth is one doing, passing more than 2 MiB of arguments to a program?

Simplest example is:

    mv * /some/place/

in a folder with a lot of files.

Also, if Python had a limit of 2M bytes in e.g. a list, then all hell would break loose. Yet, Bash is some people's Python, under some circumstances (and in Bash you're more likely to run into this problem).

Reply on news.ycombinator.com