It's important to understand that functions like execve(), which are used to spawn processes, are upstream dependencies of dynamic memory functions like malloc(). It's hairy to have low-level functions in your system depend on other functions that are higher level than them. For instance I've been in situations where malloc() failed and the LLVM libcxx abort handler depends on malloc(). POSIX also defines execve() as being asynchronous signal safe, which means it isn't allowed to do things like acquire a mutex (which is necessary to allocate unbounded dynamic memory).
On a few occasions I wished that Python by default limited the max length of its lists to, say, 100 million elements to avoid bad bugs that consume memory and trigger swapping for few minutes before been killed by OOM. Allocating such amount of memory as plain Python list, not a specialized data structure like numpy array, is way more likely indicate a bug then a real need.
There is already a hard limit, the amount of memory before the OOM killer is triggered.
So why can't the limit for shell args be the amount of memory before the OOM killer is triggered as well?
It can. Just set RLIMIT_STACK to something huge. Distros set it at 8mb by default. Take it up with them if you want the default to change for everyone.
I mean we wouldn't even need to have this discussion if the limit was at the memory limit.
Imagine having this discussion for every array in a modern system ...
I think I, and the parent commenter, are just pointing out how arbitrary the limit is. It can't hurt to question stuff like this every once in a while.
There's always a limit. People only complain when it actually limits them. Most open source people have never needed to glob tens of thousands of files. If you want to feel better, POSIX says the minimum permissible ARG_MAX is 4096, and with Windows ARG_MAX is only 32767 characters.