Well the difference between the pre-bathroom door handle and the post-bathroom door handle (opposite sides of the same door, presumably) is that one of them is much more likely to have been touched by someone who may have just been to the toilet without washing their hands.

You wouldn't be that worried about touching the door on the way in especially if you knew you were going to be washing your hands in a moment.