Hacker News Reader

No chance I'd be able to run this on an 8GB M1 Mac mini then? Looking to hook this up to an IRC bot for a bit of fun with my friends and have this spare Mac lying around.

The 7b model loads on a lot less than 8GB RAM, so you should be good to use it on an 8GB M1 Mac Mini. It takes very little time to set up, so it's worth trying to find out.

Yeah I'll get this set up when I'm home from work! Perhaps the trickiest part for me is going to be hooking it up to IRC, but that's the fun part!

7B will be fine I have heard people getting llama working on a nexus 6

Thanks!

There are efforts like https://github.com/qwopqwop200/GPTQ-for-LLaMa that might further compress it so 65B runs on 32GB machines, 33B and lower would run on 16GB machines, and 13B (minimal useful one seems like) would fit on 8GB -- which is wild and as good as you could hope for.