I don't know about other "anti-maskers," but for me, it's about the precautionary principle [1]. If a "basic protective measure" is not foolproof, then maybe the downsides of such a measure outweighs the benefits, and before we decide, collectively, to change our habits, maybe we should think hard about those downsides.
Yes, there are downsides to masks: reduced oxygen, keeping captured bacteria close to your face for long periods, etc., made worse by the fact that people, by and large, do not rotate or clean their masks properly. In other words, experiments in a lab that study the effectiveness of masks when used properly do not match the reality of how people actually use masks.
In other words, in science, all factors matter. [2] And pretending otherwise is disingenuous.
[1]: https://en.wikipedia.org/wiki/Precautionary_principle [2]: https://gavinhoward.com/2020/08/in-science-all-factors-matte...