There are also special designed masks for this. I like the lung plus better, as the inside of a mask gets snotty and wet and glasses don't fog with the lungplus. But others I know use the mask variants and are happy. They aren't as dorky looking I guess, at least not after everyone started wearing masks heh. Note that having a buff ("neck gaiter") above your mouth isn't really comparable to a mask. It will just get wet and cold when breathing hard through it during exercise and is not really airthight so cold air still slips in other ways. Try a proper mask/lungplus designed for this, I can recommend it.
[0]: http://lungplususa.com/ not the prettiest webpage I've seen, but the product works