Hacker News Reader

Comment by wwwtyro | Hacker News Reader

Is there any reason to believe that the first open source chat GPT clone won't consume most mindshare, a la stable diffusion?

View on HN

There are already open source LLMs with comparable parameter counts (Facebook's OPT-175B, BLOOM), but you'll need ~10x A100 GPUs to run them (which would cost ~$100K+).

I suspect a big part of why stable diffusion managed to consume so much mindshare is that it can run on ordinary consumer hardware. On that point, I would be excited about an open-source RETRO (https://arxiv.org/pdf/2112.04426.pdf) model with comparable performance to GPT-3 that could run on consumer hardware with an NVMe SSD.

They aren't as good as davinci-003, though. There is no open source model competitive with GPT-3.5 yet.

GPT-J-6B is said to be also of comparable performance in certain areas to GPT-3 and can be run (inference) on a RTX3090.

https://huggingface.co/EleutherAI/gpt-j-6B

One can in theory run even 175B bloom with just a modern multicore CPU, 32gb of RAM and 2TB of nvme storage. There is a library called accelerate witch with some slight modifications allows one to run in cpu/storage only mode models that don't fit in memory. Of course it takes a looong time to do inference, but one can at least have a taste.

The biggest bloom I personally have run on cpu only in this fashion is 7B. It requires 4x7B of RAM plus some. On my hardware it tends to use all 32GB RAM and about ~4GB of storage during inference. At the moment I believe there is still a limitation of the smallest layer fitting in memory at once. This is why I haven't tried bigger bloom, but I believe there are ways to overcome it. Once this problem is resolved one should be able to use the same tech to use GPUs with less vram (like my 2070 with 8GB) for parts of larger models.

Deployment is the significant barrier with these models. They have far more parameters than stable diffusion/DALLE2.

It all ends up on HuggingFace anyway so that's where the smart money is heading.

on top of the hardware requirements ($10s of thousands of GPUs are needed for something a language model as big as gpt-3) there's also a lot of work involved in RLHF models like chatgpt. you need to pay people to write and review thousands/tens of thousands of responses for training. see 'methods' here: https://openai.com/blog/chatgpt/

LAION’s Open Assistant trying to address that with gamified crowdsourcing.

https://github.com/LAION-AI/Open-Assistant

My theory is that OpenAI is preying on venture capital, and they don't care who wins long term. As long as they're first to get the freshest ideas on the biggest computers, they can secure a large sum of money.

And do what with it?

Sounds like SNL's "First Change Bank" skit:

- You give us a dollar, we'll give you four quarters!

- People ask us how we make money. The answer is simple: _volume_.

I see what you're saying, but the truth is that for a moment there the bank had $2 on its books.

Reply on news.ycombinator.com