Hacker News Reader

GPT-3 is the name for the architecture, but there are a few different versions/sizes. The OpenAI version that impressed us all was ~170B parameters, this is far smaller.

To go from 2.7B to 170B parameters will need more than just a few config tweaks. There's a whole bunch of hacks and tricks needed to coax a model to train at that scale, the Eleuther version is almost guaranteed to fail out-of-the-box.

View on HN

It's worth noting that the GPT-3 paper did train models with more sane sizes (e.g. 1.5B) as a point of comparison. I am surprised/annoyed they never released them though.

It's because OpenAI sells them for profit. The "Ada" model is the same size as the larger of these two EleutherAI models.

Huh, I was wondering what the size of the non-davinci models were; guess that make sense.

It's still telling that a "small" GPT-3 model can risk cannibalizing a larger model.

Ada is 2.7B, Babbage is 6.7B, Curie is 13.0B, and DaVinci is 175B. The new one they announced last month is in the 20-50B range I think, not totally sure though.

Reply on news.ycombinator.com