Hacker News Reader

Comment by lxe | Hacker News Reader

Got the ops-6.7b chatbot running on a windows machine with a 3090 in mere minutes. The only difference was to install the cuda pytorch `pip install torch==1.13.1+cu117 --extra-index-url https://download.pytorch.org/whl/cu117` just like in stable diffusion's case.

It performs as expected:

    Human: Tell me a joke
    Machine: I have no sense of humour

    Human: What's 2+5?
    Machine: I cannot answer that.

View on HN

6.7b is pretty small, no? Do you even need offloading for that on a 3090? I'd be curious to see what's needed to run opt-30b or opt-66b with reasonable performance. The README suggests that even opt-175b should be doable with okay performance on a single NVIDIA T4 if you have enough RAM.

It is entirely possible to run 6.7B parameter models on a 3090, although I believe you need 16 bit weights. I think you can squeeze a 20b parameter model onto the 3090 if you go all the way down to 8.

Hey. So did anyone try doing it with AMD cards ( I know Nvidia seems preferable now )?

Looks like it might be no bueno on google colab for now, chatbot.py takes prompts via input() too rather then a command line argument.

Could it work on Google Colab?

Reply on news.ycombinator.com