Hacker News Reader

Comment by fpoling | Hacker News Reader

There is always a limit. An explicit value versus implicit depending on memory size of the system have a big advantage that it will be hit sufficiently often so any security vulnerabilities will surface much earlier. Plus it forces to use saner interfaces to pass big data chunks to a utility. For that reason I would even prefer for the limit to be much lower on Linux so the commands will stop to assume that the user can always pass all the settings on the command line.

View on HN

Would you advocate to put a hard limit on Python lists too?

There is already a hard limit, the amount of memory before the OOM killer is triggered.

So why can't the limit for shell args be the amount of memory before the OOM killer is triggered as well?

It can. Just set RLIMIT_STACK to something huge. Distros set it at 8mb by default. Take it up with them if you want the default to change for everyone.

I mean we wouldn't even need to have this discussion if the limit was at the memory limit.

Imagine having this discussion for every array in a modern system ...

I think I, and the parent commenter, are just pointing out how arbitrary the limit is. It can't hurt to question stuff like this every once in a while.

There's always a limit. People only complain when it actually limits them. Most open source people have never needed to glob tens of thousands of files. If you want to feel better, POSIX says the minimum permissible ARG_MAX is 4096, and with Windows ARG_MAX is only 32767 characters.

On a few occasions I wished that Python by default limited the max length of its lists to, say, 100 million elements to avoid bad bugs that consume memory and trigger swapping for few minutes before been killed by OOM. Allocating such amount of memory as plain Python list, not a specialized data structure like numpy array, is way more likely indicate a bug then a real need.

It's important to understand that functions like execve(), which are used to spawn processes, are upstream dependencies of dynamic memory functions like malloc(). It's hairy to have low-level functions in your system depend on other functions that are higher level than them. For instance I've been in situations where malloc() failed and the LLVM libcxx abort handler depends on malloc(). POSIX also defines execve() as being asynchronous signal safe, which means it isn't allowed to do things like acquire a mutex (which is necessary to allocate unbounded dynamic memory).

Reply on news.ycombinator.com