I think they get pulled down by the combination of atmosphere and gravity. As you pointed out, gravity alone is not sufficient. But also, if there's only an atmosphere but no gravity, they don't get pulled down.
Not quite true. For example quite a few orbits around the moon are unstable and you will end up crashing into the surface. This is due to essentially all celestial objects not being spherical.