GPT4 Has 1T Parameters

Yeah, I'm highly suspicious too. Even the arxiv article from the MS researchers doesn't have specifics about the # of parameters in GPT-4.

Yeah seems spotty. Especially considering recent "chinchilla scaling" laws suggesting training set size is generally the current bottleneck, the mileage llama/alpaca gets out of 7b/13b, the huge inference cost of 1T, etc.

Sam altman said it had 1T parameters in the Lex Fridman podcast

I've had significant improvements with getting more complex boilerplate code out of it, it practically wrote a flask webapp with a HTML/JS UI for me with minimal corrections. I was pasting the entire codebase into each new context, giving my instructions, and getting complete and working modifications back. Anecdotally, GPT3 was more error-prone on simple coding tasks, though GPT4 has its slipups.

Entirely anecdotal: The more I use it, the more it seems somewhat... sporadically drunk? My present use-case is character-acting, I can 'feel' when the conversation has hit up against some sort of filter or barrier because characters lose track of what is happening, fail to follow basic instructions, and the logic of the conversation breaks down (who knows what, what has been exchanged, etc.), even in fairly short and well-structured conversations. The effect is greatly heightened if the conversation broaches uncomfortable topics, or if the character GPT is playing has a personality other than 'helpful assistant'.

Honestly, the content is worse than early GPT 3.5 - as they built protections against jailbreaking and imitation, they also necessarily protected against role-playing. The characters were initially wilful and human, now they are very docile and bot-like, and stop making sense if their personality contradicts the Chatbot's.

I'll be using open models for content generation in the future, and ChatGPT for other semantic layers.

I think they learned from Microsoft Tay what happens if you train AI online without a curation process.