Hacker News Reader

Comment by joshuahedlund | Hacker News Reader

Is there a Stable-Diffusion-esque open-source GPT yet? Given the incredible pace of advances in the image space this year, and my (perhaps naive) assumption that text generation is less complex and less resource-intensive than image generation, I'm hoping we'll get something similar and surprised that we haven't yet.

View on HN

There is, called "GPT-NeoX", or its TPU-based predecessor GPT-Neo. However, even running inference on these models is much, much harder than Stable Diffusion -- the GPT-NeoX-20B weights for GPT-NeoX requires a minimum of two GPUs with 24 GB of VRAM each to simply run inference, never-mind training or fine-tuning.

I believe there are some tricks for cutting down the VRAM requirements a bit by dropping precision at different points, but the gist is that these big text models are actually quite a bit more resource intensive than the image models.

Interesting that they're saying they have a target of replicating GPT-3 with 175B parameters, but since the RETRO paper it should (in theory?) be possible to replicate GPT-3 with a tenth of the parameters. Are they not planning to use a retrieval system, is it difficult to adopt their system for retrieval, or is their readme out of date?

Where can i find more information about running GPT-NeoX? Is it covered in the paper? Or is there a forum, HOWTO or Wiki somewhere?

You could follow EleutherAI's official guide: https://github.com/EleutherAI/gpt-neox You could also use a hosted service that proposes GPT-NeoX like https://nlpcloud.com or https://goose.ai

There are multiple open-source GPTs, but GPT-3 is absolutely massive - larger than the image models actually! So, unfortunately, text generation is probably even more complex and resource intensive than image generation (especially to train). Additionally, in image generation, we appreciate the creativity of solutions, but in text generation the creative solutions seem like utter nonsense.

I guess my intuition is based on the file size of text being so much smaller than images, but I guess that doesn't really map to the complexity of generating it. Fascinating!

I think large language models are still in their infancy. The models are extremely sparse, but we don‘t have the tooling yet to deal with these kinds of structures efficiently. Your intuition might be right in a future, maybe.

If you think about the space both models are covering from a rate-of-failure perspective, it kind of makes sense that images end up being a bit easier than text: both text- and image-models can output results that look plausible at first glance, but when you analyze both outputs further there are a lot more gotchas in parsing meaning within language than there are in pixel placement within an image.

Actually, I'd argue that images generated by SD have much more flaws than texts produced by GPT-3. GPT-3 (at least, the full model) is quite capable of writing stuff that "make sense", but most eye candy results generated by SD are cherry picked, the others are simply crap.

That's kinda the point here I think. GPT-3 is trained on much more data than SD and contains much more knowledge. SD is actually similar in size to GPT-2.

Image models of the same size as GPT-3 should be much more impressive, the difference will probably be quite large like the difference between GPT-2 and GPT-3.

Ask SD to write a step by step guide to do something and it will create an image that looks kinda like some instructions, but the contents will be nonsense.

An image model of the size of GPT-3 could probably do this task quite well in many cases.

Image models needs much better language understanding to get to the next level also though, so probably multimodal models may make more sense. Maybe feeding web pages rendered as images to an image model could give interesting results.

Google recently released the Flan-T5 models that are very efficient, they rival GPT-3 and can run on a normal desktop. You can try them out.

https://huggingface.co/google/flan-t5-xxl?text=Translate+to+...

Emad Mostaque, CEO of Stability AI (behind Stable Diffusion) has said they are training language models that they plan on releasing. IIRC, though, even using all the most up-to-date techniques like 1-byte parameter quantization and Chinchilla scaling learnings, it will still probably be on the order of magnitude of 64GB or something, so not quite usable for most people locally yet, unless you happen to have a very beefy multi-GPU machine at home.

GLM-130b is available and uses 8-bit quantization, was trained on 400 billion tokens, and runs on 4x3090, which, of course, is why all the cheap Vast.AI instances are gone :)

Seems unlikely Emad would release

There are smaller models that you can freely play around with that work in roughly the same way. If you're working on a fairly regular computer some reasonable options are GPT2 or GPT-Neo. These can both perform inference on your local CPU if you have 8GB or more of RAM.

They are much less powerful that GPT-3, but they can still be fun for simple text generation or NLP tasks. You can play around with one of the smaller GPT-Neo models that should fit in RAM if run locally here:

https://huggingface.co/EleutherAI/gpt-neo-1.3B

That page includes instructions to run this locally in Python.

As others mentioned, there are larger models available, but they tend to be expensive to setup and use as an individual.

BLOOM[0] is another 176B multilingual open-source model whose weights were published.

However, running it locally requires 8 A6000 GPUs. The GPU cost alone is roughly $37K.

Also, there is the concern of performance. The Chinchilla[1] paper indicates that most model sizes trained so far received too little token training (or were too big) to reach an optimal performance per energy expenditure. Furthermore, the InstructGPT[2] and more recently Flan[3] papers show that various finetuning techniques achieve significant gains, without which the result is a bit meh.

The dark secret of GPT-3 is that the model available on the OpenAI website is heavily fine-tuned, even compared to the InstructGPT paper.

[0]: https://huggingface.co/bigscience/bloom

[1]: https://arxiv.org/abs/2203.15556

[2]: https://arxiv.org/abs/2203.02155

[3]: https://arxiv.org/pdf/2210.11416.pdf

Great. Soon, we'll have scripts written in GPT non-sense with actors and other imagery generated by stable diffusion. I'm guessing they'll still charge $15 prices at the theater for it.

Reply on news.ycombinator.com