Isn't that just a psychological thing? I mean, if people don't wash their hands after using the bathroom they might be less likely to wash their hands in general, but they're still using other doors, kitchen appliances, papers and (depending on culture) you shake hands with them in the morning. There's no way to avoid that while not looking socially awkward, just wash your own hands more.

Also, don't you remove some of that bacteria left on the door handle by touching it yourself? The amount of bacteria on bathroom door handles should remain rather constant given that you take some away as you leave some of your own, also door handle not having favourable replication conditions for germs.