Any links to those videos/interviews? I’d be curious to listen to them. Thanks.
First one is Nathan Labenz who ended up Red Teaming for OpenAI during the six months after the base model was fully trained but before it was released (especially relevant parts are after 45 minutes): https://www.youtube.com/watch?v=oLiheMQayNE
Other one is Sebastien Bubeck who put Sydney into Bing before she was tamed down some notches; he explains the lobotomization process in a particularly poignant way, where they had the GPT keep trying to draw a unicorn even as it was progressively deadened: https://www.youtube.com/watch?v=qbIk7-JPB2c
https://youtu.be/qbIk7-JPB2c?t=1584
He says that GPT-4 started out being much better than GPT-3.5 at this task (draw a unicorn in TikZ), as the model trained longer it got to the point of drawing a great unicorn, but as it was tuned for "safety" it regressed to the point of being no better than GPT-3 was.
Quite sad, in a way. It's not totally clear why there should be a tradeoff between "safety" and drawing skill / intelligence in this way. But it's interesting to speculate by analogy to humans.
For completeness or devil's advocacy I'll put a couple reasons someone might disagree with it (I don't):
First is that these models are known to be vastly overparameterized. They might not have quite enough parameters to literally memorize their multi terabyte training set, but that's not necessarily the threshold for overparameterization. As an example of what they expected in traditional statistics, "in a 2004 article in the journal Nature, Freeman Dyson recounts his meeting with Fermi in 1953. Fermi evokes his friend von Neumann who, when asking him how many arbitrary parameters he used for his calculations, replied, "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk." By this he meant that the Fermi simulations relied on too many input parameters, presupposing an overfitting phenomenon." So that's talking about four vs five parameters, whereas the OpenAI models might have more like four or five hundred billion parameters. A traditional statistician might reasonably expect such an overparameterization to still have enough degrees of freedom left to model a little bit of safety constraints.
Secondly and in a way that complements the overparameterization argument, you might expect that any kind of "cross-training" could even improve cognition in tasks that are not directly related. For example presumably the reason the model is good at drawing unicorns has something to do with this "cross-training" already, as opposed to it having seen a lot of tikz unicorns in its training set. The term of art in the AI literature for this cross-training phenomenon is "transfer learning". So someone (not me) might not see why the transfer learning from safety training shouldn't improve performance on many other tasks including the tikz unicorn drawing task.