You could check whether the email passed SPF, DKIM, and DMARC checks. Google has certainly implemented these for their SMTP servers. If all of the tests pass, you can be pretty certain that the sender is the owner of the address.
Though, admittedly, it would be easier for a layman to verify ability to reply to an email than to verify all those features, so point taken.