Not just the physically damaging effects of smoking, but I also wonder if being a heavy marijuana user is correlated with other features that are causal for infection risk, e.g. unhealthy diet, inconsistent sleep, lack of exercise.
FYI, I've smoked weed every day for 44 years straight.
Today, I am at, or near the peak, of my physical health (180 lb, 6'2, daily swimmer), I sleep like a baby and wake up every day at 8 am. And I swim every day.
Swimmers, in general, are optimistic and far more tolerant of misery, and you haven't accounted for that bias.
Also there are plenty of other athletes who swear by marijuana usage, you aren't even unique in that! My point is that the stereotype of a heavy marijuana user is very different from you, and that said stereotype could provide an alternative causal mechanism that we need to be accounted for in research.