> All this will do is drive up the cost of housing as remote workers compete for it, driving out the locals.
Rural America is huge. If every single person in New York city got themselves a suburban house, they'd take up 0.1% of the United States (about the area of two counties).
Housing is expensive in cities because 10 million people want to live in the same square mile. The US, however, has 4 million of those square miles. (Everyone in New York City could move to California and have 10 acres of their own. I did the calculation with the intent of showing that everyone in the country could have their own square mile, but that's not true -- I didn't realize how big a square mile actually is.)
Remote workers probably aren't going to raise the cost of living that much. It is certainly possible that they could all move to the same place (i.e. turn Aspen, Colorado into New York City), and that would be a problem. But if people randomly move into rural areas, there just aren't enough people buying 1 million dollar apartments in the same place to "gentrify out" farmers.
I'll also point out, that remote work doesn't necessarily imply that it's possible to move to the middle of nowhere. You still need a good Internet connection, and many people want to live closer than an hour's drive away from a grocery store or hospital. I work remotely but I'm happy to live 3 minutes away from Manhattan. There are factors other than commuting to work that account for where people live.