CarperAI announces plans for the first open-source “instruction-tuned” LM

I wonder how the number of parameters of this will compare to GPT-3 - will there be different sizes? Their stated goal is to make this available to academics/independent researchers/startups, and I think with a 175B parameter model, not having the trained model itself isn't necessarily the biggest hurdle for most people (having hardware with x hundred GB memory is - well maybe not for startups.)

Although there seems to be hope:

>In prior work, OpenAI found that the outputs from models trained with RLHF were preferred to those from 100x larger models trained without human feedback.

> CarperAI will release a chinchilla-optimal large language model

Next-level terminology

for now I'd checkout T0 or bloom, lots of models can be instructed already.

I created https://text-generator.io and have a whole bunch of examples on there of instruction following style prompts

I would urge them to build something good that fits on 24GB maybe 48GB VRAM, otherwise there will be slow opensource progress. I need Codex alternative like that desperately to fix programming.

I've been waiting for this ever since they trained BLOOM and even GPT Neo. Without instruction tuning they are almost worthless. If all you want is classification then it's better to use BERT, but if you need the generative part then you have to scale up to these big guys.

What I need is the instruction following skills of GPT-3, running on a single (large) GPU, and not having to pay by token.

Can anyone explain what’s new here? I couldn’t quite follow the article.

I wonder if this relates to DeepCTRL that came out recently.