Hacker News Reader

This is not something that will be possible to run locally.

If you had 1 bit per parameter (not realistic), it would still take ~100 GB of RAM just to load into memory.

You could technically dynamically offload the RAM overload to disk but this would probably be too slow?

I mean, theoretically if you can get the model weights onto disk then you should be able to do the computation - but it might takes days or months on commodity hardware. It would also require creating a system that can do this and I doubt there is much demand.

Reply on news.ycombinator.com