If you ask an AI if "P=NP", it will certainly give an answer. It has to, because the answer is either "yes" or "no".

If you ask it to then prove its assertion, it will then certainly output the tokens that look like a plausible proof. It has to, as it is programmed too.

This "proof" might even be thousands of pages of very technical looking and professional sounding jargon.

It's not a real proof though, and as an artifact it is 100% useless to both the field of mathematics and humanity.