Wouldn't countries just adjust their olympic spend such that it becomes profitable to host them rather than lose interest altogether?
I remember the lead up to the Sydney Olympics and no-one seemed to want it - then it arrived and it was an amazing 2 weeks long party and everyone loved it.
Beijing on the other hand was the converse - everyone was expecting it and looking forward to it and then it came and the city went in to virtual lockdown.
You never know which way it's going to go.