Of course it is. That is the outcome of unsupervised learning.

It doesn’t have a sense of what’s true and false, what’s right and wrong.

It has learned to predict the next word very well and the prediction probability distribution was later tweaked with human feedback and automated test feedback.

They’ll train it to not predict these words as much (or basically at all) when this is asked. But a very large part of the model will not be touched by these interventions, so it will continue to predict text as it has learned to.