Hacker News Reader

Comment by mehrdadn | Hacker News Reader

Translation: This is a euphemism for saying Linux blatantly violates the language specification with no remorse.

(Yes, I realize Linux is the kernel, etc.)

Linux is a kernel, not a C runtime so that's not really relevant. You say you realize that but then what are you arguing for exactly? Arguably you could complain that the glibc (or whatever libc you're using) is not working around that by, for instance, scrubbing the pages in malloc() to force the kernel to allocate memory.

But even then I'm not convinced that anything here is non-standard, at worse maybe we're in a bit of a grey area. As long as the kernel maintains its smoke and mirrors whether and how it allocates memory is irrelevant from the point of view of the standard. The C language has a rather simplistic memory model, it doesn't impose a lot on the implementation.

Now the problem occurs when the program attempts to access virtually-allocated memory and the kernel realizes that it can't find any physical memory to map it to. In this situation several things can happen but in general the process will be killed. Is it against the standard for the OS to kill a program for arbitrary reasons? I can't imagine why. It could also freeze the program, waiting for more memory to become available. Again, not against the standard as far as I can tell. Or maybe kill some other program to free memory.

If you have some specific part of the C standard in mind please do tell, I always find these language lawyering arguments interesting, somehow.

> If you have some specific part of the C standard in mind please do tell

See here: https://news.ycombinator.com/item?id=20145604

Also note the POSIX standard:

Upon successful completion with size not equal to 0, malloc() shall return a pointer to the allocated space. If size is 0, either a null pointer or a unique pointer that can be successfully passed to free() shall be returned. Otherwise, it shall return a null pointer and set errno to indicate the error.

In neither case is there any provision for returning a non-null pointer to anything other than an allocated block of memory of at least the given size.

Thank you for your reply but I still don't buy it. As far as the program is concerned it is returned a memory block, what this "memory" is effectively behind the scenes is none of the standard's business. As long as the implementation manages to maintain the illusion it's perfectly fine AFAIK. The problem is when this breaks down and the kernel realizes that it can no longer maintain the masquerade. If at this point it did something stupid like map this block nowhere or remap something that's already been allocated and let the program continue like this, then yes that would be a clear violation of the contract. But it does no such thing, it kills the program instead. If there's no program then there's no problem. At no point did a single C instruction get executed in an environment that did not maintain a coherent memory model. >In neither case is there any provision for returning a non-null pointer to anything other than an allocated block of memory of at least the given size. A pointer to virtual memory is a pointer to memory. The rest is an implementation detail. If, when dereferenced, the kernel issues an order to Amazon for more RAM and waits for it to be installed to resume the execution, that's none of the C standard's business.

> If, when dereferenced, the kernel issues an order to Amazon for more RAM and waits for it to be installed to resume the execution, that's none of the C standard's business.

We're not, and we never were, debating the situation where dereferencing the non-null pointer returned by malloc succeeds but takes long time due to your Amazon order. We've been talking about the situation where it fails. malloc is not allowed to return a non-null pointer to a memory block that cannot be written to. Linux does it anyway, and in doing so blatantly violates the standard.

That is my point, it doesn't fail. Either the kernel finds out a way to map the memory and it succeeds, or it kill the program and the instruction never runs. Code that doesn't run can't violate the standard. When the code is allowed to run all the invariants are guaranteed to be respected. If I write this code:

    char *b = malloc(2);
    if (b == NULL) {
      return 0;
    }
    
    b[0] = 'A';
    b[1] = '\0';
    
    printf("%s\n", b);
    
    free(b);

The standard tells me that if the malloc succeeds then the following code, if allowed to run, will display "A" on stdout. The C standard cannot and does not guarantee that a C program can't be interrupted however. For the sake of the argument we could imagine a kernel that instead of killing the program freezes it indefinitely on disk waiting for RAM to be available. It's functionally the same thing. As long as the kernel doesn't let code run with broken invariants it's fine. This is completely outside of the scope of a language standard to define.

Or, to try one last time from a different direction, if you consider that the C standard mandates that accessing memory returned successfully by malloc has to be successful and I happen to press ^C when that happens in a program, should the kernel refuse to kill the program? This is obviously absurd, but it's effectively the same thing: the kernel reacts to some external state and decides to terminate the program.

Okay this is a far more reasonable argument but I'm not convinced it's right. The standard does define normal and abnormal program termination. It also defines <signal.h>. SIGINT addresses the keyboard case (or the implementation can provide another signal). SIGABRT, SIGTERM, etc. are raised upon termination. For the keyboard case, then the program would be made aware, and it can indeed ignore the Ctrl+C request if it desires. That's perfectly normal and nothing absurd. For other types of termination, the other signals are raised. But in this case the program is terminated before any signal is raised at all. Now, as a practical matter I would argue the hosted environment should still be able to kill the program in the face of user request simply because nobody wants a host that's the slave of the program even if it's against the standard, but this is going far, far, far beyond that. It's happening at the request of neither the user nor the program; it's just happening because the host feels like it. Now that's both a violation of the standard and an uncool one at that!

P.S. I don't think indefinite hold is "functionally the same thing" as termination. A caller system(), for one, would need to return in one case, but not the other.

If a tiny, strictly conforming ISO C program obtains a tiny block of memory from malloc (like a few hundred bytes) and crashes when trying to initialize it, that is almost certainly a non-conforming ISO C implementation. The reason is that the implementation cannot support even one single program that exercises each of the implementation limits.

See 5.2.4.1 in http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf

However, it's true that not any old instance of this malloc problem demonstrates such a nonconformance. If it happens in a large program that has allocated gobs of memory, then no.

Basically if the system is low on memory that it can no longer support the execution of a small C program with modest memory use, then it becomes nonconforming.

However, the mere property that memory can be doled out by malloc which might later not be used doesn't make it ipso facto nonconforming.

Moreover, a system with any kind of memory management (including management that earnestly reports null for "out of memory") can be come a nonconforming C implementation if it is low on memory.

Allocated "space", i.e., virtual memory space. Malloc allocates virtual memory space, not physical RAM.

C has no notion of virtual memory. If memory is allocated for you that means you can read and write to it.

There is no guarantee offered by the standard that you must be able to read and write to memory allocated by malloc(). What if the memory was allocated and just a split second later, the physical RAM burnt out due to overheating? How is the standard supposed to guarantee reading and writing to it then?

Both the hardware burning after memory allocation and lack of availability of physical memory after memory allocation are outside the scope of the standard. The standard says nothing about them. A C program can fail in these scenarios without violating the standard.

> Linux is a kernel, not a C runtime so that's not really relevant.

Maybe it's more accurate to say that Linux is a kernel that assumes many features and semantics of the C runtime. But it certainly seems more deeply intertwined than you're willing to address here.

> Linux blatantly violates the language specification with no remorse.

We may be able to argue that it doesn't. ISO C says in 1. Scope, this:

This International Standard does not specify

[...]

— the size or complexity of a program and its data that will exceed the capacity of any specific data-processing system or the capacity of a particular processor.

It seems that these weasel words have an interpretation that can be bent around overcommitted memory allocation.

No it wouldn't. They clearly included the ability to return null to allow for when the data exceeds the capacity of the system. There is no provision for returning a pointer to memory that cannot be used.

You might think, but that's not how how conformance (of an ISO C implementation to the standard) works.

For an implementation to be deemed conforming, it just has to be demonstrated to successfully translate and execute one program that tests each of the minimum implementation limits.

See 5.2.4.1 in http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf

Only if no such program can be found can we then conclude that the implementation is nonconforming (and if the reason for not finding such a program is this overcommit issue, then we can blame that issue).

Your Linux system is indeed nonconforming if it is so low on memory that, for instance, no C program can allocate a 65535 byte object (that can be actually initialized, and used: a real object). Basically the memory situation has to be so severe that it takes the implementation below the minimum limits, whereby we can clearly demonstrate nonconformance.

Which part of the specification does this violate?

Like, every part? It doesn't actually allocate the space requested, it doesn't indicate failure by returning NULL (or by any other reliable means for that matter)...

I had a look, and the C standard doesn't actually say anything about indicating failure. Here is all it says about malloc:

> The malloc function allocates space for an object whose size is specified by size and whose value is indeterminate.

> The malloc function returns either a null pointer or a pointer to the allocated space.

I think it's certainly debatable whether overcommitting while lazily allocating counts as actually allocating.

> I had a look, and the C standard doesn't actually say anything about indicating failure.

> The malloc function returns either a null pointer or a pointer to the allocated space.

Uhm, either malloc returns a pointer to the allocated space, or it returns null. I don't see what's so complicated about this. There's no provision for "return a non-null pointer to unallocated space".

The malloc() function requests memory from the kernel using mmap(). If mmap() returns successfully, it means that the kernel is saying that the memory is allocated. So from malloc()'s perspective the memory allocation has been successful and it must return a non-null pointer to the caller. The standard does not require malloc() to distrust what the kernel said and try to actually write to that memory to double-check if there is any physical memory mapped to it or not. The standard also does not impose anything on the kernel. I see no section of the standard being violated here.

Section 7.22.3.1:

The order and contiguity of storage allocated by successive calls to the aligned_alloc, calloc, malloc, and realloc functions is unspecified. The pointer returned if the allocation succeeds is suitably aligned so that it may be assigned to a pointer to any type of object with a fundamental alignment requirement and then used to access such an object or an array of such objects in the space allocated (until the space is explicitly deallocated). The lifetime of an allocated object extends from the allocation until the deallocation. Each such allocation shall yield a pointer to an object disjoint from any other object. The pointer returned points to the start (lowest byte address) of the allocated space. If the space cannot be allocated, a null pointer is returned. If the size of the space requested is zero, the behavior is implementation-defined: either a null pointer is returned, or the behavior is as if the size were some nonzero value, except that the returned pointer shall not be used to access an object.

Section 7.22.3.4:

The malloc function allocates space for an object whose size is specified by size and whose value is indeterminate.

The standard literally could not care less if there's a kernel underneath. It doesn't care how malloc is implemented or what the kernel, if any, looks like "from its perspective". The perspective that has relevance is that of the caller of malloc. From your own quotes:

> The pointer returned if the allocation succeeds is suitably aligned so that [...]

> If the space cannot be allocated, a null pointer is returned.

If you use mmap and "from your perspective" that's considered success then it's your perspective that's faulty. The standard is literally telling you right here^ there are 2 possibilities: either you allocate the space and return a pointer to the space, or you don't and you return null. There is no third option of "space cannot be allocated but you return non-null anyway". That's quite literally the end of the story.

The space IS allocated, it's just that this space (virtual memory) is not backed by physical RAM.

Try telling that to the C standard. It has no notion of virtual or physical memory. If it's allocated that means you can read/write to it, end of story.

> It has no notion of virtual or physical memory.

Exactly! That's why once virtual memory is allocated, malloc() is allowed to consider the operation successful. The standard does not care at all whether it is virtual memory allocation or physical memory allocation. It is completely unspecified in the standard what sort of memory must be allocated. So no spec in the standard is being violated by returning non-null pointer for virtual memory allocation.

You can't be serious about "every part". This definitely does not violate section 1.1 which only talks about the scope of the standard. It also definitely does not violate 1.2 which talks about what is not within the scope of the standard.

So once again, can you cite the exact section number from the standard that you think is being violated here?

See reply to sibling comment.

There is a deep irony in folks choosing to implement one of the few things the C standard DOES clearly define by papering over it with an alternative memory model.

Ironically, it's a memory model that greatly reduces the performance of modern systems, and als impacts its safety.

> [...] Linux blatantly violates the language specification [...]

We can't expect a kernel to be aware of all language specifications.

Yes, I know that in this case C is both the program's and the Kernel's language in this example but even then. Languages go through iterations (versions) and a Kernel can't be required to obey all languages which might run under it.

If you're suggesting Linux did not intend its malloc implement the corresponding function in the C standard then you're just lying to yourself. That's exactly why it's there and that's exactly what it's used for.

> If you're suggesting Linux did not intend its malloc implement

Linux (the kernel) doesn't have malloc. It's part of the C library, which is a completely separate project. What Linux implements is brk and mmap.

Playing loose with the word Linux here. Play along a bit maybe?

The C spec actually states at the beginning, "This International Standard does not specify ... the size or complexity of a program and its data that will exceed the capacity of any specific data-processing system or the capacity of a particular processor;"

Yes and that's why malloc can return null.

Reply on news.ycombinator.com