I had malloc fail recently, on a machine with low rlimits. Here's an example on FreeBSD 8.2:
#include <stdlib.h>
#include <stdio.h>
#include <sys/resource.h>
const int SIZE = 64*1024*1024;
int main() {
char *buffer = NULL;
struct rlimit r;
/* Limit all memory to 1024 bytes */
r.rlim_cur = 1024;
r.rlim_max = 1024;
if (setrlimit(RLIMIT_AS, &r) == -1 ) {
perror("setrlimit(RLIMIT_AS) error\n");
exit(EXIT_FAILURE);
}
if ( (buffer = malloc( sizeof(long) * SIZE) ) == NULL) {
perror("malloc() failed\n");
exit(EXIT_FAILURE);
}
free(buffer);
printf("Malloc worked\n");
return 0;
}
Here's the output:
%cc fail.c
%./a.out
malloc() failed
: Cannot allocate memory
If malloc failure testing is important to you, then you'll need some way to replace your use of the system malloc/free with special versions of your own. (Eg, through function pointers, or LD_PRELOAD.) Your own versions can be crafted to insert faults in just the right spots.
It's tedious work.