It's worth noting that the GPT-3 paper did train models with more sane sizes (e.g. 1.5B) as a point of comparison. I am surprised/annoyed they never released them though.
It's still telling that a "small" GPT-3 model can risk cannibalizing a larger model.