I think the downside of not wearing a mask is, in fact, normalising not wearing a mask in the eyes of people who do not take into account that it’s not worn for a reason and because there is no one near me. Thus I try to put mask on when I see someone on the street even if they are far away.
I think not wearing masks again will be easy to normalise. For example, Hong Kong got hit by the first SARS and everyone wore a mask for a time, and then when it was over all was back to normal and masks quickly became rare.