Hacker News Reader

Comment by gaogao | Hacker News Reader

Meta has released the model weights for OPT-175B, which is used in the paper. There's also a lot of full release LLMs from other labs on the way as well.

View on HN

While OPT-175B is great to have publicly available, it needs a lot more training to achieve good results. Meta trained OPT on 180B tokens, compared to 300B that GPT-3 saw. And the Chinchilla scaling laws suggest that almost 4T tokens would be required to get the most bang for compute buck.

And on top of that, there are some questions on the quality of open source data (The Pile) vs OpenAI’s proprietary dataset, which they seem to have spent a lot of effort cleaning. So: open source models are probably data-constrained, in both quantity and quality.

OPT-175B isn't publicly available, sadly. It's available to research institutions, which is much better than "Open"AI, but it doesn't help us hobbyists/indie researchers much.

I wonder when we'll start putting these models on the pirate bay or similar. Seems like an excellent use for the tech. Has no one tried to upload OPT-175B anywhere like that yet?

It could go on the clear net since trained weights aren't subject to copyright.

It’s fun to think about a few billion weights being the difference between useless and gold.

Looking at my bank account I can relate :)

Are there any that perform anywhere close to GPT-3?

Stable Diffusion was behind DALLE, but has since surpassed it. Don't hear anyone talking about DALLE anymore.

Reply on news.ycombinator.com