These models / nets / whatever are much "smarter" (loaded term) than we think, we just don't know how to plug-in properly yet.
"We are not interested in the fact that the brain has the consistency of cold porridge." — Alan Turing
"We are not interested in the fact that the brain has the consistency of cold porridge." — Alan Turing
Simply asking the LLM in two separate contexts the same question but from opposing perspectives, then in a third context asking it to analyze both responses and choose the most neutral and objective take, you wipe out any "(dis)agreeableness" bias and dig closer to a deeper, more nuanced synthesis of a given topic. This paper is just taking this idea to the next level.
This isn't really possible with RLHF alone unless you train the LLM to often give two opposing perspectives, which would get tiring.
One more reason to be wary of pushing for better capabilities.