Hacker News Reader

Would love an answer on this too. It would be even better not just to try using this, but also be able to run it locally, something that has been impossible for GPT-3.

View on HN

This is not something that will be possible to run locally.

If you had 1 bit per parameter (not realistic), it would still take ~100 GB of RAM just to load into memory.

You could technically dynamically offload the RAM overload to disk but this would probably be too slow?

I mean, theoretically if you can get the model weights onto disk then you should be able to do the computation - but it might takes days or months on commodity hardware. It would also require creating a system that can do this and I doubt there is much demand.

Does it look like it would be possible to run locally?

Reply on news.ycombinator.com