I feel like we need a name for this error. Pretraining fallacy? Coding ability is all about RL.
Remember when they finetuned an LLM to write buggy code and it also started saying humanity should be exterminated and so on? It knows good and evil and is trained to select one.