Hacker News Reader

Comment by afhof | Hacker News Reader

So once you've allocate more memory than you have, what happens when you to to write to any of it?

I altered the program to first allocate 64G then go through and write on each page of the allocation on my 16G machine, dropped my swap partition.

After suitable screwing around to keep gcc from optimizing away the writes, it goes a bit like this…

  jim@gvdev:~$ ./fail
  Before
  Allocated 1 GB
  Allocated 2 GB
  ...
  Allocated 64 GB
  After
  Writing in 0 GB
  Writing in 1 GB
  ...
  Writing in 15 GB
  ... rather long delay here...
  Write failed: Broken pipe
  upstairs:~ jim$    <----- bad news here, notice the machine.
  upstairs:~ jim$ slogin gvdev.XXXXX.net
  ssh: connect to host gvdev.XXXXX.net port 22: Network is unreachable

… so now I have to get back in "going out" clothes, go downstairs, and drive over to gvdev and reboot it.

Kinda OT, but out of interest, why do you use 'slogin' instead of 'ssh'? Just old habits dieing hard from 'rlogin'?

Yes.

So, when there is no memory and no swap space available, instead of crashing, it zombifies whole system? This is weird.

The linux Out Of Memory killer would have activated (if not disabled). This program derives a 'score' for each program, based on a number of statistics, and kills the process with the highest score. Although it usually makes the right choice, sometimes it can kill innocent programs.

It was worse than a dead sshd. Looks like something kernel-ish died badly. There was one more cron message in the syslog after this, but then nothing until I rebooted. Console unresponsive, no ping answering.

  Nov  2 21:29:33 gvdev kernel: [15115318.546376] fail[17150]: 
      segfault at 7fffe9cbc1e0 ip 00000000004006fe sp 00007fffe9cbb1d0 
      error 4 in fail[400000+1000]

This is a problem with virtual machines vendors such as Linode where the configured swap space is usually tiny by default... It's much better to provide more swap (At least as much as RAM size) and see the performances start to deteriorate as a sign of alarm of overcommit, instead of having random processes killed.

Always good to have your sshd on auto-respawn in case OOM makes a bad choice ;) Learned this in a similar scenario.

That depends on what you mean "more memory than you have". If it's more physical RAM than is currently available, not necessarily a problem - some other processes can be pushed out of RAM into swap. If it's more than that, some pages can be discarded because they're backed by binaries on disk. If it's more than that, some processes can be killed. If it's more than that, really bad things happen.

The problem is that there's no good way to differentiate between a process that merely wants more VM that it'll never touch and a process that wants actual RAM. In theory, every fork() should double your VM requirements - in practice, almost all are immediately followed by an exec() and the previous allocations are irrelevant. Do you fail fork() because you can't guarantee you can back every modified page in the new process? If not, why should you fail malloc() because you can't guarantee you can back it? There's no correct answer, and Linux lets you control which answer you want via /proc/sys/vm/overcommit_memory .

Actually on solaris, the fork does OOM a java VM. So while this is a pragmatic solution on linux, its also a false promise which will paper over some transient memory requirements while leaving a huge gap in regards to realistic resource expectations. Unfortunately this problem is magnified when using virtual OS instances, and shitty hardware.

Everything's ok until you try to write to more memory than is available. At that point, either your program segfaults, or Linux's Out-Of-Memory (OOM) Killer is invoked and kills a random process (not necessarily the offending process).

(not necessarily the offending process)

It's terrible about this with PostgreSQL. The OOM Killer tends to thump the postmaster, not the offending backend.

You should instruct the OOM Killer to not kill PostgreSQL.

Memory is a global resource, even if you didn't overcommit the process getting the allocation failure is not guaranteed to be the misbehaving one, it's just the unlucky process that did the first allocation after all the memory was gone (this is also why handling allocation errors in your application is usually impossible, even without an overcommitting kernel).

It's not something that's easy to develop a general solution to. Maybe the process that's using a lot of RAM is not runaway, and is mission-critical. Maybe it is a runaway process and something else is mission-critical.

The most 'fair' way to go about it is to just kill a random process. That skits the whole issue.

Zero-Fill on Demand - All new pages map to a single zero "frame" in physical memory. In this case, you don't get any memory until you write to the page, at which time the OS steps in and allocates you a page of all zeros

EDIT: this means if the kernel can't allocate more memory when you write then the kernel has a decision to make; kill you, or kill someone else to get more memory.

Reply on news.ycombinator.com