We know for a fact that GPT-4's RLHF makes the model worse by incurring an "alignment tax." Microsoft Research and others had access to GPT-4 pre-RLHF and all reported that it was far more capable then and that it lost capability as RLHF/Alignment training checkpoints came in.