Yes this makes sense and it's probably the reason.
For completeness or devil's advocacy I'll put a couple reasons someone might disagree with it (I don't):
First is that these models are known to be vastly overparameterized. They might not have quite enough parameters to literally memorize their multi terabyte training set, but that's not necessarily the threshold for overparameterization. As an example of what they expected in traditional statistics, "in a 2004 article in the journal Nature, Freeman Dyson recounts his meeting with Fermi in 1953. Fermi evokes his friend von Neumann who, when asking him how many arbitrary parameters he used for his calculations, replied, "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk." By this he meant that the Fermi simulations relied on too many input parameters, presupposing an overfitting phenomenon." So that's talking about four vs five parameters, whereas the OpenAI models might have more like four or five hundred billion parameters. A traditional statistician might reasonably expect such an overparameterization to still have enough degrees of freedom left to model a little bit of safety constraints.
Secondly and in a way that complements the overparameterization argument, you might expect that any kind of "cross-training" could even improve cognition in tasks that are not directly related. For example presumably the reason the model is good at drawing unicorns has something to do with this "cross-training" already, as opposed to it having seen a lot of tikz unicorns in its training set. The term of art in the AI literature for this cross-training phenomenon is "transfer learning". So someone (not me) might not see why the transfer learning from safety training shouldn't improve performance on many other tasks including the tikz unicorn drawing task.