One of the problems is the pure mechanics of how SMTP works: anyone can send an email with any "from" address, and filtering on people/domains in the "from" field is basically pointless if you want to catch spam.

What Google is likely doing is checking the domain of the originating IP in the SMTP "envelope", but that also gets tricky with outsourced email services or internal IPs.

I do not use Gmail, so these are just wild guesses, but I do run my own mail server and frequently get my email not showing up for people.

Problem with SMTP bounces is that it may take a week for the final bounce to show up in my inbox (because again, that's how SMTP protocol is designed, to expect nodes to be down and retry a number of times).