If you get the same result over and over again, it means the model is more overfit to a certain result. It does not mean the result is correct.
Thank you, this is one of the most understood 'facts', especially regarding "prompt hacking/jailbreaking"
> model is more overfit to a certain result
From their communications, a massive amount of effort was put into making sure the model followed the system prompt. One might claim "overfit as a feature".