Mortgage lenders will put huge premiums on mortgages in the town in the future because of "default by eminent domain" risk, if the government pulls freddie/fannie mae it'll be even more expensive. Basically they're robbing future mortgage owners in order to pay current ones.
Worse yet, other towns with similar profiles might have their risk profile increased too, so they'll get extra premiums because of Richmond's behaviour without getting any of the benefits.
So while it might seem like they're just taking the money from pension funds and other investors, in the longer term they'll be taking far more from future mortgage owners.