Hacker News Reader

I think the main advantage here is you COULD run it, even it it takes a while. That is a step up from current model limitations which require ram or vram to hold the model.

I think this lays some groundwork for running a 400B model on a 3090/4090 or even smaller GPU. If you can get a huge model like that running on a single gpu even if the mean time per token is in the seconds, that's acceptable for many use cases.

If this same technique can be used to extend context windows in addition to token autocomplete, that would be great in it's own regard.

Hopefully work like this continues as throwing a ton of vram at a model should be regarded as a performance optimization not necessarily a requirement.

It's already technically possible to run huge models locally when you don't have the RAM/VRAM needed - llama.cpp can 'mmap' the model from disk.

Of course an nvidia 4090 has a memory bandwidth of a 1000 GB per second; a CPU like the i7-13700K has a memory bandwidth of 90 GB per second; and a high-end NVMe SSD might only have read bandwidth of 10 GB per second.

So in approximate terms, an LLM and quantisation level that can produce 10 tokens per second on a 4090 will produce 1 token per second from RAM and a token every 10 seconds from SSD.

I haven't been able to get llama.cpp's mmap logic to work on macOS

Yeah mmap and leaving the caching up to the OS or storage drivers instead of model-informed placement is not going to yield the same results. Observing data access patterns, pre-empting inference data requirements and understanding hardware latency when distributing the model can yield some pretty significant results, as that same approach does in other domains. This is adjacent to compiler optimization.

It would be very meta to use AI to observe these access patterns and distribute the model accordingly based on usage, so that it can optimize placement for your given context domain.

> That is a step up from current model limitations which require ram or vram to hold the model.

Current? Apple recently published a neat paper on how they optimise for both inference (cpu/gpu) and memory use:

  Our method involves constructing an inference cost model that takes into account
  the characteristics of flash memory, guiding us to optimize in two critical areas:
  reducing the volume of data transferred from flash and reading data in larger, more contiguous
  chunks. Within this hardware-informed framework, we introduce two principal techniques.
  First, “windowing” strategically reduces data transfer by reusing previously activated neurons,
  and second, “row-column bundling”, tailored to the sequential data access strengths
  of flash memory, increases the size of data chunks read from flash memory. These methods
  collectively enable running models up to twice the size of the available DRAM, with
  up to 4x and 20x increase in inference speed compared to naive loading approaches in CPU
  and GPU, respectively.

https://news.ycombinator.com/item?id=38704982

Similar but I think the apple approach here requires model modification whereas afaik OPs solution works with the model verbatim. I could be wrong as I haven't looked into the code but given the specificity of the first article regarding hardware and model, I would assume that.

> I think the main advantage here is you COULD run it, even it it takes a while.

I mean, you COULD run it before as well, even if you don't have enough RAM or VRAM, by using something like `zram`. It'd probably be even slower (and border-line usable, depending on the use case), but it's not impossible to get things to run.

Zram compresses part of the data in ram, right? Can an LLM be compressed?

Not really. The weights are mostly random numbers.

Do you think this could allow distributed inference only, or opens the door for distributed training of the model? Democratization of the models is in part hampered by the total compute a single person or small group can make use of, but if a project like folding@home, but for training large models is possible, it could change the game somewhat.

> I think the paper makes the case that you could probably recruit say your 30 graphics workstations to do much faster inference without just nailing your LAN bandwidth, though.

Could be a big deal if it allows cluster of smaller GPUs to compete with a single large VRAM GPU.

Unfortunately I’m a few months of date - which is an eternity in LLM inference techniques - so I’m not sure what current state of distributed inference looks like.

llama.cpp supports splitting work across multiple nodes on a network already

it essentially just copies a chunk of the model to each one, works well for situations where each machine has limited vram

Any pointers to RTFM / llama repo for that ? I could not find anything on a cursory look. Thanks in advance !

https://github.com/ggerganov/llama.cpp/tree/master/examples/...

you run that on the remote nodes

Yeah, I think these methods could be baked into llama.cpp or some other higher up the toolchain python library or what have you. They shard out each layer (ish?) to the edges, and recombine that layer inference at the master node, while the outside edges load up their next bit if they need to; I would guess the devil is in the details for all the possible tensor types and architectures (for instance, how shall we implement skip layers?).