That makes it hard for a city in a free society to want to be the host. The costs are very real but the benefits are elusive. Probably the last time the Olympics were profitable to the host city was the 1984 Los Angeles games and that was because Los Angeles was able to drive a hard bargain because nobody wanted to host the Olympics after the terrorism at the 1972 games.
Some people have suggested we might be better off if the Olympics were held in the same place(s) every year, maybe in Greece.