Most of the studies have been done in high risk, high exposure situations. The high concentration and repeated exposure over extended periods of time greatly increase the chance of infection. This does not reflect the lower concentrations and shorter duration of most civilian exposures. It generally takes more than a single virus to actually result in infection.
Since there are indications that wearing masks does give you some protection and does give significant protection to those around you, why not accept the minor inconvenience and wear one?