Only if the servers aren't using SMTPS. Which in practice many don't but there really is no reason why that cannot be changed.
If yourserver.com is connecting to friendsserver.com on port 25... there is some metadata. I mean if three-letter-agencies are monitoring the internet they'll figure out what just happened.