There are downsides, like when the government threatens violence against its citizens for not complying with a preventative measure with mixed efficacy at best.
What government has threatened violence in regards to mask mandates?
I guess anything that is fined or outlaws is eventually "enforced by violence" but it's still a silly argument. If a store asks you to leave, and you don't, and the store calls the police to arrest you, you aren't being arrested for not wearing a mask. You're being arrested for not leaving when asked.
Do you really think the businesses are enforcing masks for the sake of public health? I encourage you to ask them -- 9 times out of 10 they will say they are fearful of the government fining them and shutting down their business. Trust me, I ask them every time they ask me to leave and it's always the same answer. If they resist the government, it will eventually result in violence.
So are you not wearing a mask? If not, why not?
There are dozens to hundreds of regulations governing the activities of retail establishments.. the requirement to enforce mask mandates is only one.