You're basically just gish galloping at this point and there's no need to respond to you any more. All your points about swap are irrelevant to the discussion. All your points about disk cache are irrelevant to the discussion. You have a very, very, very incorrect understanding of how operating system kernels work if you think mmap() is just a less efficient read():
> Every word of that is nonsense. The disk is attached to a serial bus. Even if you're using fancy nvme "disks" there's a queue that operates in a (reasonably) serial fashion. The reason mmap() is referred to zero-copy is because it can reuse the block cache if it has been recently downloaded -- but if the data is paged out, there is absolutely a copy and it's more expensive than just read() by a long way.
Please do a basic web search for what "virtual memory" means. You seem to think that handwavey nonsense about disk cache means that read() doesn't copy the data into your working set. You should look at the manpage for read() and maybe ponder why it requires you to pass your own buffer. This buffer would have to be something you've malloc()'d ahead of time. Hence why you're using more memory by using read() than you would using mmap()
> You only have so much storage, this is life, but I can tell you as someone who does operate a 1tb of ram machine that downloads 300gb of logfiles every day, read() and write() work just fine -- I just can't speak for python (or why python people don't do it) because i don't like python.
You should definitely learn what the mmap syscall does and why it exists. You really don't need to use 300gb of RAM to read a 300gb log file. You should probably look up how text editors like vim actually work, and why you can seek to the end of a 300gb log file without vim taking up a bunch of RAM. Maybe you've never been curious about this before, I dunno.
Try making a 20gb file called "bigfile", and run this C program:
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <inttypes.h>
#include <unistd.h>
int main(int argc, char *argv[]) {
char *mmap_buffer;
int fd;
struct stat sb;
ssize_t s;
fd = open("./bigfile", O_RDONLY);
fstat(fd, &sb); // get the size
// do the mmap
mmap_buffer = mmap(NULL, sb.st_size, PROT_READ, MAP_PRIVATE, fd, 0);
uint8_t sum;
// Ensure the whole buffer is touched by doing a dumb math operation on every byte
for (size_t i = 0; i < sb.st_size; ++i) {
sum += mmap_buffer[i]; // overflow is fine, sorta a dumb checksum
}
fprintf(stderr, "done! sum=%d\n", sum);
sleep(1000);
}
And wait for "done!" to show up in stderr. It will sleep 1000 seconds waiting for you to ctrl+c at this point. At this point we will have (1) mmap'd the entire file into memory, and (2) read the whole thing sequentially, adding each byte to a `sum` variable (with expected overflow.)
While it's sleeping, check /proc/<pid>/status and take note of the memory stats. You'll see that VmSize is as big as the file you read in (for me, bigfile is more than 20GB):
VmSize: 20974160 kB
But the actual
resident set is 968kb:
VmRSS: 968 kB
So, my program is using 968 kB even though it has a 20GB in-memory buffer that just read the whole file in! My system only has 16GB of RAM and swap is disabled.
How is this possible? Because mmap lets you do this. The kernel will read in pages from bigfile on demand, but is also free to free them at any point. There is no controversy here, every modern operating system has supported this for decades.
Compare this to a similar program using read():
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <inttypes.h>
#include <unistd.h>
int main(int argc, char *argv[]) {
int fd;
struct stat sb;
ssize_t s;
char *read_buffer;
fd = open("./bigfile", O_RDONLY);
fstat(fd, &sb); // get the size
// do the read
read_buffer = malloc(sb.st_size);
read(fd, read_buffer, sb.st_size);
uint8_t sum;
// Ensure the whole buffer is touched by doing a dumb math operation on every byte
for (size_t i = 0; i < sb.st_size; ++i) {
sum += read_buffer[i]; // overflow is fine, sorta a dumb checksum
}
fprintf(stderr, "done! sum=%d\n", sum);
sleep(1000);
}
And do the same thing (in this instance I shrunk bigfile to 2 GB because I don't have enough physical RAM to do this with 20GB.) You'll see this in /proc/<pid>/status:
VmPeak: 2099928 kB
VmSize: 2099928 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 2098056 kB
VmRSS: 2098056 kB
Oops! I'm using 2 GB of resident set size. If I were to do this with a file that's bigger than RAM, I'd get OOM-killed.
This is why you shouldn't read() in large datasets. Your strategy is to have servers with 1TB of ram and massive amounts of swap, and I'm telling you you don't need this to process this big of files. mmap() does so without requiring things to be read into RAM ahead of time.
Oh, and guess what: Take out the `sleep(1000)`, and the mmap version is faster than the read() version:
$ time ./mmap
done! sum=225
./mmap 6.89s user 0.28s system 99% cpu 7.180 total
$ time ./read
done! sum=246
./read 6.86s user 1.72s system 99% cpu 8.612 total
Why is it faster? Because we don't have to needlessly copy the data into the process's heap. We can just read the blocks directly from the mmap'd address space, and let page faults read them in for us.