Another important aspect of the RL process is that it's fine-tuning with some feedback on the quality of data: a 'raw' LLM has been trained on a lot of very low-quality data, and it has an incentive to predict that accurately as well, because there's no means to effectively rate a copy of most of the text on the internet. So there's a lot of biases in the model which basically mean it will include low-quality predictions in a given 'next token' estimate, because if it doesn't it will get penalised when it is fed the low quality data during the training.
With RLHF it gets a signal during training for whether the next token it's trying to predict are part of a 'good' response or a 'bad' response, so it can learn to suppress features it learned in the first part of the process which are not useful.
(you seem the same with image generators: they've been trained on a bunch of very nice-looking art and photos, but they've also been trained on triply-compressed badly cropped memes and terrible MS-paint art. You need to have a plan for getting the model to output the former and not the latter if you want it to be useful)