Of course if you live in a very polluted city the harm is much more pronounced: https://ig.ft.com/sites/urban-cycling/assets/pollution-world...
Including increased accident risks, particulate, and chemical pollution together is a very different risk profile vs comparing all benefits vs a subset of downsides.
That said, you can filter out most particulates and there is little reason not to take this precaution.
Also, many cars include high quality air filtration which really does help.
But again, just buy a mask and deal with the actual problem.
However, like a bike helmet an air filter is at most a minor inconvenience.
PS: I suspect re-breathing a tiny amount of CO2 probably has a measurable impact over a longer race. (Don't have data on this though.)