Why MMAP in llama.cpp hides true memory usage
twitter.com
twitter.com
Short version: the claim that llama could use much less memory by memory-mapping the model data file was wrong. It's just that the Linux utilities for memory use don't count memory-mapped file area.
[1] https://news.ycombinator.com/item?id=35393615
edit: softened some of my claims after reading more updates from over the weekend.
echo 3 > /proc/sys/vm/drop_caches
And because reading the file probably allocated file backed pages for it anyway unless you used O_DIRECT (and profiled it first.)
There's also wired/mlock() memory which is "worse" than regular memory because it can't be swapped/evicted at all.
* Fewer context switches among user space / kernel syscalls
* No need to copy _modified_ data into the swap. The behavior for read only data doesn't change
> * Fewer context switches among user space / kernel syscalls
This is not necessarily true. There's lots of cases where it's actually slower.
Note that each page fault to read an mmaped page is also a context switch from user space into the kernel. The kernel entry/exit paths for syscalls and page faults have much in common.
Mmaped pages can in principle use fault-ahead to reduce the number of page faults when sequential access is detected. An equivalent reduction is available for read syscalls by reading larger blocks at a time.
In practice, mmap files are faster in some scenarios compared with read syscalls and slower in others. Same with O_DIRECT reads, those are faster in some scenarios and slower in others.
The new MMAP system would not require you to load the whole model from disk up front, but rather would allow you to load and immediately forget each layer of parameters as you do a forward pass through the GPT architecture. And if you had just tried to use normal swap paging to do this without MMAP, you'd essentially be doing 2x the loading work, since the page for the first layer likely got swapped out as you loaded the rest of the model into "memory."
Of course, as soon as you want to generate a new token, you'd need to reload all the pages of the model parameters. Every token generated would take as long as it would if you were loading the model from scratch.
But for a classification problem where you only need to generate one token, and you need or want to do this in a serverless environment where you'd need to load from disk anyways? I think you'd be able to do that workflow with significantly less RAM in the same amount of clock time, now.
Wouldn't make sense to keep most of the model in GPU (as much as it fits) and only load the remaining layers on each pass?
If that is the case then perhaps a high end consumer system with max pcie 5 nvme bandwidth and multi gpus could do inference on large LLM models.
I keep seeing references to these absolutely massive 48GB+ gpu memory sizes and those are extremely expensive. If more but smaller memory gpus can work that could open up a much larger range of hardware and individuals to run inference, especially if efficient load/unload of weight layers from raid pcie5 nvme devices is feasible.
The original LLaMA-65B on fp16 weight is 130G. There are no NVIDIA GPU with 130G memory.
As for why they don't use int8/int4: would you try to optimize your code before you confirmed it can actually run?
- max PCI lanes for more GPUs. AMD cpus and boards probably better
- PCIe 5 nvme. ideally enough to saturate bus
- max GPUs with max memory. memory more important than FLOPs probably.
for a given budget one tries to get as much as possible of above.
How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?
PCIe 5 NVMe: That's completely irrelevant for inference, as you really want to keep all weights on the GPU VRAM. It only speed up the initial weight load from host to device.
max GPUs with max memory: Yeah. For personal use (batch size = 1) VRAM size AND VRAM bandwidth is more important than FLOPs. See https://orenleung.com/is-chatgpt-175-billion-parameters-tech.... For production it's a tradeoff.
> How long before a consumer component machine for less than $10K can do inference for a gpt 3.5 turbo quality model?
$10k is a lot. You can run LLaMA-65B on 2x4090, won't be fast enough for production though. But given that people happily run LLaMA-13B on their MacBooks I guess we are okay with generation being slow. And personally I don't think gpt-3.5-turbo is that big, it should be around or even smaller than LLaMA-65B.
Uh two months ago? LLaMA-13B is GPT 3.5 Turbo quality and runs locally on my Android phone's CPU at acceptable speeds.
You can even run LLaMA-65B (which far surpasses GPT 3.5) faster than GPT 3.5 Turbo with two $200 24GB Nvidia Tesla P40 cards, since in 4bit the model is only 39GB with no output quality loss. Total cost $400 plus some junker used PC with two spare PCIe 4 x16 lanes.
Are P40s only $200!
If your information is accurate that is incredible and great tip. I’ll start on building such a rig asap.
This means it does page eviction on memory pressure and also results in deduplication as mmap'd memory of the same file can be used by any process that has access to it.
If the data access is sequential in nature (or at least has some form of locality in function of time) it could end up using less memory indeed, even if the whole file ends up being read during the training process, because unused pages will be evicted by the OS to make room for needed ones.
What is interesting to me here is that it's such a marvel to these ML guys. It's a fundamental OS facility and looks like using mmap() instead of using read() on malloc()'d memory should have been the strategy here from the start.
Swap works by letting the kernel write a memory page to the swap file/partition. After it has done that, the memory is marked as backed by a filesystem (i.e it's no longer anonymous), and the region of physical RAM can be repurposed for something else. With mmap'd files, the memory is always backed by a filesystem; there are no writes, and the physical memory can be discarded at any time.
The difference is between reading a file and memory mapping (mmap'ing) it.
If you read a 1TB file into memory you use 1TB of disk and 1TB of physical memory. If you then access that data it's as fast as RAM because that's where it is.
If you mmap a 1TB file you use 1TB of disk and 1TB of virtual memory. If you then access that data it may be mapped into virtual memory but not actually be in RAM. This triggers a page fault, at which point the correct page is loaded from disk to physical memory, and handed back to you.
The key observation is that the amount of physical memory occupied by the mmap'd file is much smaller than the entirety of the file unless you access almost all of it.
If your filesystem supports holes, this can also be useful for writes: it's possible to map files vastly larger than physical disk space, but so long as the actual number of places written to is quite small you won't run out.
The combination is very useful for datastructures because you basically don't have to care about data being extremely sparse until that data is also getting quite large, which means you can use cheap/fast approaches to indexing, etc.
The operating system is free to load and cache the file into memory in whatever way it wants, so for large memory mapped files it'll often try to use all available memory to cache as much as possible. If another program needs memory, the operating system will simply lower the amount of memory available to the memory mapped file for caching. This is extremely useful for databases, since it greatly simplifies both how to persist the data along with how to load and cache the persisted data.
This all comes with a big caveat however. The less memory you have, the more file accesses occur (similar to your pagefile when you're thrashing), which can dramatically slow down your memory operations.
tldr; it lets you designate a file to use as a region of memory.
It saves a copy on read, in exchange for a bit more overhead configuring the processes address space up front.
> "The fundamental operation of mmap is to add new entries to the page table of a process, and the precise properties of those entries are heavily dependent on what the arguments to mmap are.
> When you mmap a regular file, you're essentially adding an entry to the page table that shares the data with the kernel's filesystem cache."
Now, understanding this requires some understanding of an OS's virtual memory function, and what "page tables" are. Those are what the OS use to track a process' memory, whose granularity is in "pages" (historically 4kB on Linux, though others use larger granularity such as 16kB).
mmap() has a a lot of flags [2] that affect the properties of those mapped pages. It is the swiss-army knife of memory management on Linux.
Some of those properties allow you to share memory with other processes (MAP_SHARED | MAP_ANONYMOUS), or just allocate memory (MAP_ANONYMOUS) or, by default, map an (open) file specified by the `fd` argument.
(fun fact! On linux, when you malloc(), you don't actually get memory, just an IOU from the kernel. Only when you access that memory, ie accessing those memory pages, does the kernel actually make the effort of allocating you that memory.)
It seems the intuitive way to implement read if there's a complex memory subsystem that does all these great things anyways.
The OS somehow detecting that argument to read(2) is page aligned block of uncommited meory and transparently creating MAP_PRIVATE mapping from that seems like nifty idea, but userspace tends to not have page-aligned buffers that often (see the aforementioned 0x10 offset for one reason why) and applications that intentionally allocates IO buffers with some particular alignment do so in order to bypass the block cache for particular IO patterns and are exactly the kinds of applications that tend to use mmap(2) for IO where that does not matter.
Hmm. You're right about glibc, but macOS for instance reliably returns page-aligned pointers for large allocations. I wonder how Windows behaves.
Well, first of all, it's not: malloc may have previously stored its internal structures in that memory so it may be dirty. Second, AFAIK if you ask malloc() for a gigabyte, it'll internally call mmap() on an anonymous file and will give you its result.
But from what I understand there may be nothing in POSIX forbidding the kernel to do this optimization, just kernel authors decided against it (which is understandable).
I can think of one edge case not mentioned by others, though.
One of the main benefits of mmap is that the data can be paged out while not used. But when you try to page the data back in, the attempt may fail, or it may return different data than was originally there. This is particularly likely with network filesystems or external storage. With mmap, these situations result in, respectively, the process receiving SIGBUS, or the contents of memory changing out from under it. The former is usually a crash (unless the program installs a signal handler), and the latter can be worse than a crash, if either program logic or compiler optimizations make assumptions about memory staying consistent when not written to. A program using mmap is opting in to that risk. A program using malloc and read is not.
That said, it may be reasonable to trust that the system’s main disk(s) (however you define that) won’t misbehave this way, and limit the optimization to files mapped from them. After all, if swap is enabled, the same potential risks exist for any memory that’s swapped out. Even without swap, if the root partition starts failing, the system is not going to stay up for long.
Or, the optimization could be enabled for other disks, but with the caveat that the pages would be swapped to the swap file/partition rather than paged out normally.
Still, the current problem with mmap is that it's buggy on some operating systems according to https://www.sqlite.org/mmap.html, so I guess it will cause more frustrations until AGI takes over the world :)
When a read call completes, you know you've successfully read the data from the underlying device. If you have swap disabled you know you can read and write from that memory quickly.
With mmap'd data, a memory read/write can trigger a page fault and I/O, and if you encounter an error like a network filesystem not responding in time, the program sees a segfault.
`mmap` signals the kernel that you want to use the data someday. You `mmap` it, it automatically handles fetching and throwing unused bytes away.
`read` signals the kernel that you want to do something with the data now.
You `read` it, you need it in the memory now.
> why the kernel can't see that it's unused allocated memory
Only a in-process garbage collector would really know what part of your memory is unused. The kernel can't really know for sure.
https://manybutfinite.com/post/anatomy-of-a-program-in-memor... [1]
You can't really understand mmap/sbrk without understanding virtual memory and process space layout.
[1] Images are broken. Just open https://static.duartes.org/img/blogPosts/kernelUserMemorySpl... and go to advanced and "Proceed to static.duartes.org" to workaround their https issues. The duartes.org host is owned by the blog author. Refresh the blog article and images should load now.
>"This thread explains how the mmap feature of modern operating systems can be used to reduce memory usage when running large deep learning models. It also explains how...
Yet I wonder, if this was just another of those humans imitating a trustworthy AI...
> The way mmap handles memory makes it a bit tricky for the OS to report on a per-process level how that memory is being used. So it generally will just show it as "cache" use. That's why (AFAICT) there was some initial misunderstanding regarding the memory savings
Edit: except embedding weights but those are not the problem.
Obviously if you're swapping because you don't have enough memory to hold the model in RAM, the mmap version is going to be much faster, since you don't need to swap anything out to disk but just discard the page and re-read from disk if you need it again later.
The issue is typically that that initial load involves some sort of transformation - parsing, instantiating structures, etc. If you can arrange it so that the data is stored in the format you actually need it in memory, then you can skip that entire transformation phase.
I don’t know if that’s what’s been done with llama.cop though.
Not necessarily memory savings, but the improvements here are that it will run on computers with less ram because it can page to disk.
Not necessarily the number reported (since you do need to load chunks into ram), but still lower.
file (DISK) to process active pages (RAM) to paged out virtual memory if saturated (elsewhere on DISK)
Using mmap typically goes something like:
file (DISK) to process active pages (RAM) to released and cached (RAM) to uncached if evicted (back to same place on DISK) OR back to process active pages from cache (RAM)
For the cost of fixing up some process page tables, the physical memory pages necessary can be brought back from the cache rather than read from disk. It's an orders-of-magnitude performance savings.
One persons "paging the same memory requirement from disk to RAM" can be someone elses "requiring less memory/RAM".
It's part of the program's working set but measuring that is a completely different story.
However, such image mapping technique is not always suitable for data-intensive workloads due to page thrashing. If it works for llama.cpp then it can be considered a huge success.
... for some definition of "modern".
(Also you’re thinking of TOPS-10 — TOPS-20 and TWENEX were developed in the 70s. But your heart is in the right place!
Its quite obvious that the author "dumbs down" all the information for a very non-developer audience, so I assume that gives some leniency. If you tell people that we have had things like mmap() for decades, they may start asking too many questions about why every piece of software underperforms so horribly below what was possible decades ago.
Bit of a rant, but I feel that we lose a little bit of potential every time a developer calls operator<< on a std::istream in a loop.
This is, as an aside, the reason why I don't comment my code often (I personally dislike docblocks that do nothing but obvious explanations) - but whenever I think hard about a thing I always record it in a comment with the problem, my reasoning, and my rational for choosing the solution used... i.e. don't comment on the what, comment on the why.
You may consider removing the rubber only when you can adqueately explain to me why it is there
Instead you have to do backflips to load data that doesn't fit in RAM.
Also just to show how buggy mmap is, it's disabled in SQLite by default for example:
https://www.sqlite.org/mmap.html
Even now the windows version is buggy, but the great thing is that after it's fixed in llama.cpp, the open source community can just copy the solution.
I don't know of many interactive utilities that were known for fast startup time _because_ of the use of mmap, but now we have one, which means many more are coming :)
- depending on concurrent accesses mmap can just sigbus on you (e.g. if the mapped file is being truncated)
- mmap simply does not work with virtual filesystems, and will blow up on large files on 32b systems (windows also has further limitations on mmaps)
- depending on workload mmap may not be faster than regular reads and memory buffers, ripgrep will mmap when working on just a few files, but will use normal buffers for large file counts, because when you start reusing buffer you amortise allocation costs which you can't amortise when creating and destroying mappings
- not only that but mmap/munmap are also globally blocking on the process, so in multithreaded processes it's very bad to map/unmap a lot, you can stall your own application
- the semantics of mmap on crash are also somewhat risky, in that the OS will try very hard to sync, but that may not be desirable if the application crashes in the middle of a write
Also, I'm disappointed that VM languages don't turn SIGBUS accessing a memory mapped region into a VM exception. They could, and it wouldn't even be that hard.
For embedded databases that operate on shared file somewhere just mmap()ing that file allows you to use part of the same file as shared memory segment and for POSIX IPC and thus synchronization between otherwise unrelated processes that access the same data. One big caveat of this approach is that it depends on optional parts of POSIX that are realistically usable on Linux and Solaris and maybe some obscure systems (you can probably do the same thing on Windows NT), notably this does not work on essentially anything BSD-derived.
How to tell if a dev is in their 20s or thereabouts