Hacker News Reader

Comment by tgtweak | Hacker News Reader

I think the main advantage here is you COULD run it, even it it takes a while. That is a step up from current model limitations which require ram or vram to hold the model.

I think this lays some groundwork for running a 400B model on a 3090/4090 or even smaller GPU. If you can get a huge model like that running on a single gpu even if the mean time per token is in the seconds, that's acceptable for many use cases.

If this same technique can be used to extend context windows in addition to token autocomplete, that would be great in it's own regard.

Hopefully work like this continues as throwing a ton of vram at a model should be regarded as a performance optimization not necessarily a requirement.

View on HN

It's already technically possible to run huge models locally when you don't have the RAM/VRAM needed - llama.cpp can 'mmap' the model from disk.

Of course an nvidia 4090 has a memory bandwidth of a 1000 GB per second; a CPU like the i7-13700K has a memory bandwidth of 90 GB per second; and a high-end NVMe SSD might only have read bandwidth of 10 GB per second.

So in approximate terms, an LLM and quantisation level that can produce 10 tokens per second on a 4090 will produce 1 token per second from RAM and a token every 10 seconds from SSD.

I haven't been able to get llama.cpp's mmap logic to work on macOS

Yeah mmap and leaving the caching up to the OS or storage drivers instead of model-informed placement is not going to yield the same results. Observing data access patterns, pre-empting inference data requirements and understanding hardware latency when distributing the model can yield some pretty significant results, as that same approach does in other domains. This is adjacent to compiler optimization.

It would be very meta to use AI to observe these access patterns and distribute the model accordingly based on usage, so that it can optimize placement for your given context domain.

> That is a step up from current model limitations which require ram or vram to hold the model.

Current? Apple recently published a neat paper on how they optimise for both inference (cpu/gpu) and memory use:

  Our method involves constructing an inference cost model that takes into account
  the characteristics of flash memory, guiding us to optimize in two critical areas:
  reducing the volume of data transferred from flash and reading data in larger, more contiguous
  chunks. Within this hardware-informed framework, we introduce two principal techniques.
  First, “windowing” strategically reduces data transfer by reusing previously activated neurons,
  and second, “row-column bundling”, tailored to the sequential data access strengths
  of flash memory, increases the size of data chunks read from flash memory. These methods
  collectively enable running models up to twice the size of the available DRAM, with
  up to 4x and 20x increase in inference speed compared to naive loading approaches in CPU
  and GPU, respectively.

https://news.ycombinator.com/item?id=38704982

Similar but I think the apple approach here requires model modification whereas afaik OPs solution works with the model verbatim. I could be wrong as I haven't looked into the code but given the specificity of the first article regarding hardware and model, I would assume that.

> I think the main advantage here is you COULD run it, even it it takes a while.

I mean, you COULD run it before as well, even if you don't have enough RAM or VRAM, by using something like `zram`. It'd probably be even slower (and border-line usable, depending on the use case), but it's not impossible to get things to run.

Zram compresses part of the data in ram, right? Can an LLM be compressed?

Not really. The weights are mostly random numbers.

Reply on news.ycombinator.com