"We should all wear masks" is not based on facts all all. Science tells us how masks work, and we can interpret those facts and learn when they should be used.
But from there to everyone wearing masks at all occasions is a leap from facts to opinion. Mandatory face masks because the costs are low and the risks are negligible is pure politics. Compare with "it was even warmer during the middle ages therefore global warming is not anthropogenic". It may be based on a fact, but the facts do not support the conclusion.
Bicycle helmets is perhaps the most well known example of this. It is clear how they protect, that they are effective, and when they should be used. That does not mean mandatory helmet laws are a good idea. In fact, the places in the world with the lowest accidents per kilometre are also the ones where helmet usage is lowest! This may be surprising at first.
Face masks may cause people to relax and enter crowded spaces they hadn't otherwise. Touching the mask is a very effective way to get virus material on your fingers. Kids playing with face masks is most certainly not a good idea. And so on. It is not unlikely that there are situations where face masks will worsen the outcome.
Mandatory helmets have been much more studied than mandatory face masks, even if this is likely to change. Until then, any policy made is necessarily politicized, not objective.