The difference in effectiveness is huge, to the point where it's ridiculous to be healthy and wear a mask. Like I said, they don't put masks on surgery patients.

And don't forget the social signal part; if everyone wears a mask all the time, then we don't know who actually considers themselves to be infectious. Whereas if just the infectious people wear masks, then we know who they are and can act appropriately (being grateful that they self-identified as infectious and kept the rest of us safer, for a start).