That is understandable. It is hard to validate if an email is authentic. SMTP has no authentication built in. Gmail can't just blindly accept all emails from addresses that you have already sent an email to.
EDIT:
Look at this example: https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#...
Anyone can connect to relay.example.com and pretend to be bob@example.com.