You could do that or you could just keep it at the same facility for, say, 20 years and THEN move it. So a country could get a 20-year economic boost (hopefully, at least long-term) and at least re-use the facilities a few times
More fundimental is the question of any boom. London didnt see much, with some shops downtown instead seeing marked reductions. Vancouver barely broke even. China didnt do it for the money. And the less said about Sochi's costs the better.
Part of it is on the verge of being converted into a cement factory (https://www.theguardian.com/uk-news/2016/aug/11/olympic-park...)
Honestly believe a world without the Olympic games is a better world, if London can't find much use for them and Rio is already letting them fall into disrepair. What on earth is the point in this.
London is an exception to olympic abandonment to my mind - it's doing a lot of the success of east London on what was previously polluted industrial land.
If you where going to host 5 games then you could build a lot larger and more enduring facilities which would still be fairly new 20 years out. Importantly, you could also make some improvements after the first game when something did not work well. Worst case, I doubt maintaining unused buildings for 4 years then doing some renovation costs nearly as much as building them from scratch.
So, I don't see how using the same buildings 5 times would be a major issue. Further, a 20 year stretch in a single city would bring a stronger connection to the games as visiting the future site of the games is far more interesting than seeing where there was once greatness.
Doesn't seem to matter to the Cubs or Red Sox who both play in parks over 100 years old and are two of the most popular baseball teams. When you don't have to build them new every Olympics the facilities themselves could be much higher quality and actually be maintained.