It becomes a problem when you let FIFA bribe BRIC or even third world country politicians (or vice versa). Then you get stadiums located in deserts or rainforests without a club nearby that is even remotely capable of filling it on a weekly basis.
I like what the UEFA (Europe's FIFA) is doing here: Bigger countries get to host the European Championship on their own, smaller countries share it, lately Poland + Ukraine and Switzerland + Austria. From 2020 on the Cup will not take place in a single country anymore, instead cities will apply on their own, which is of course impractical for the World Cup or even continental cups on bigger continents.
Most in this case is maybe 5-6. Even in rich football countries like Germany, not every 1st tier club has a 40k+ stadium.
Edit: Sorry, I read your comment to quickly, thought you were saying, that there are only 5-6 stadiums that big in Germany.