It is probably based off GPT-3 with a layer on top trained for programming specifically, like what is done with AI dungeon.
I might have missed something, but that's the gist of it.
That's GPT-J-6B, to be clear. A 6-billion-parameter model is producing better output than a 300 billion parameter model, because of what I can only assume to be sheer incompetence on AI Dungeon's part. I've also used the raw GPT-3 API, and it does better at writing than either. In other words: Doing nothing would have been better than whatever they've been doing.