Boston winters sometimes make cracking open windows impractical, but if you're in a typical old multi-unit building with old steam/water centrally-controlled heating, there's a good chance that your unit is too warm anyway (though some other units might be too cold). I've heard that some of these old heating systems were even designed assuming cracked-open windows.
The main downsides I'm aware of to open windows here usually are outside noise and seasonal pollen. For the pollen, you can run an air filter and HEPA vacuum robot and/or (supposedly) install special window screen material for the pollen. If you're in a very noisy area, that sometimes means opening and closing window based on noise at the time. If outside noise at night or early morning can be bad where you are, that might mean frontloading bedroom fresh air in the evening before bed, but shutting window before bed.
I don't know about air pollution from fossil fuels, industrial, etc.