What happens when you eventually do catch an upper respiratory disease, and your immune system is no longer used to the challenges it presents?
Ah, so this is the real reason. That's fine, but it's also not really applicable to the wider population and hence isn't about what I originally asked -- which is why we would universally continuing wearing masks forever.
The answer is we wouldn't, because that makes no sense.