The direction does matter, actually. The virus particles leaving an infected person will generally be inside droplets large enough to be caught by a mask before they get a chance to evaporate and atomize. Meanwhile, the virus encountered by an uninfected person nearby will naturally be either in smaller droplets or in airborne form. Much harder to defend against.