Is that not a recipe for adversarial training, thus ensuring increasing misalignment…?
1,538 karma · joined June 6, 2013
But why would they ask the right questions without prompting? The LLMs have no intrinsic goal to research math problems, it would just be very costly to run queries without a reason to run them... arguably a human in that loop would be a mathematician, but then we do need them after all.