It depends on what you mean by 'cave'. The classic process of cave formation happens in limestone, in 'karst' areas. These aren't evenly distributed, and some countries have tons of them and others not. But there are other ways to form caves too: erosion by the sea can form sea caves, lava tubes are created by volcanic activity, and there are cave-like mines produced by humans in many places with minerals.
So you can probably find some small hole to grovel down in most places, but the good stuff is concentrated in parts of Kentucky, the Alps, the classical Karst, parts of China and so on.