It's fascinating that these models aren't aligned to produce the best possible answer they can out of the box. I wonder what might be the underlying reason for that.
There were some papers on ArXiV showing that as GPT-4 was aligned, it got measurably stupider. For example, the test where it was asked to draw a pink unicorn using a the TikZ vector art format: during training it got better and better, and then worse and worse again as it was aligned.
A friend of mine put it well: biases are just our Bayesian priors of reality. If you remove biases, then you're erasing some of those priors. Bad priors lead to bad predictions.
Secondly when you provide this additional prompt context you are giving the model more to work with including just more information about how you are going to assess the responses. It stands to reason this should give a "better" answer in the sense that it should be able to use that information to give an answer closer to what you are hoping for and it doesn't have to read your mind about what that might be.
This is no different from how humans work. For example when you take say a trig exam the question doesn't just say "evaluate tan(3pi/8)" it says something like "Give an exact value for tan(3pi/8) without using a calculator. Simplify where possible and rationalise any denominators but do not remove common factors. Show your working. " The examiner is saying they want to see you know how use the right half angle trig identities etc not just that they can't find their calculator and need you to type something in for them. If they just gave the question in the first form you wouldn't know.
Alignment work applied after bulk data training doesn't conpletely override the undesired patterns in the underlying training data, in part because all possible undesired biases in that data haven't been identified and trained against in the alignment effort.
Eg “you are a librarian, please only give accurate answers or references or you will be fired”
GPT3.5 will suddenly become extremely risk averse in its answers.
Odds are, the gpt-4 we experience is the median response across all possible chat personas.
And while GPT4 does not have a "world model", can we please move on from the stochastic parrot argument?
My inability to emit more than one word at a time (either spoken or written) does not mean I lack multidimensional planning and conceptualization; why does this line of thinking not extend to LLMs?
The thing is, the phrase "Odds are ... median response across all possible chat personas" is nonsensical in so many ways. How are you going to define chat personas? Once you do this, how do you order them to take a median? Then, why is this something that will happen with some probability? This is not a quantifiable or testable thing that can happen. It doesn't have a semantic meaning. Maybe it has an emotional meaning like "I'm excited about GPT-4", and I would agree about that! :D
Even something like a search engine doesn't necessarily give you the site you're looking for all the time