> So far LLMs will just make up bullshit rather than say they don't know

No, my experiment has nothing to do with machine learning. I am proposing we lie to the judges and tell them one is a person and one is a machine when in fact they will both be humans.

How often will the human judges fight against the question at hand and say they are both humans?

edit: in my experiment, a judge is a human who has access to two chats at the same time and we tell the human that one of the two chats is a human and the other is a machine. The judge has to decide which is which.