Good post, it perfectly captures the problem with AI. Here we have a claim that the double cover conjecture has a proof. Verified by… no one per the link.

Now imagine this proof is wrong. How would you know? Ok, think about the process in which you determine the correctness - why not do that initially?

And there it is. The problem laid bare. Ironically it reduces to the P and NP one.