Zoning is basically irrelevant - the problem is that the cost of converting an outmoded commercial building to residential is usually within the same order of magnitude as knocking it down and putting up something built for purpose. I would bet real money that there aren’t more than a handful of municipal governments that would die on the hill of maintaining existing zoning restrictions if they had viable proposals to convert vacant commercial buildings.
Those millions of acres owned by the government are mostly desert and national parks and in any event there’s no shortage of privately owned undeveloped land closer to existing cities (or hollowed out existing cities that are ripe for a renaissance).
Also, just nitpicks, but when was $5k a down payment for a house? 1955? And BlackRock isn’t a hedge fund. You might be thinking of Blackstone (also not a hedge fund, but they own portfolio companies that are buying up a lot of single-family housing and turning them into rental properties, which is evidently extremely lucrative but I find incredibly distasteful).