I haven't worked on GPT, but if I had to guess, it certainly is not a char-rnn. There are more sofisticated versions of char-rnn that use words chunks that work better (think of prefixes, etc), as well as some that actually use entire words. Theres one that isn't an rnn, that off the top of my head I cant recall what type of nn it is, but it is a list of all words with weights of intrinsic meaning (obv not manually made), combined with the traditional rnn use so grammar stays functional. I would suspect that GPT is a super huge one of these.