Not a physicist, but I would guess that the thermal mass of cold air is of little consequence. The thermal mass of all the items in the fridge would seem to dwarf that of the air.

Sure, you lose cold air when you open the fridge, but the warm air is almost instantly recooled after closing.