Correct. I was not careful enough with all the steps of my logic here.

Being able to encrypt the message proves that the party knows the secret, which if they are the only one who knows it, proves that they are who they say they are.

Indeed, the part about "which if they are the only one who knows it" is important. If you can't trust that, then security starts to break down.

You can sidestep it by instead trusting that they own a device that no one else has, like an one time password. Now, the statement gets a bit longer:

Being able to encrypt the message => the party knows the secret => (If you ask for multiple one, you can beter trust that)=> they have the otp => they are the only one who has it => they are who they say they are.