This is trivially solved with a home equity loan. (If this proposal were broadened, non-recourse home equity loans for purposes of paying this tax would need to be expanded.)
You're selling part of the upside. That's the point of the land value tax. Someone who owns a $1mm house from a $100,000 purchase didn't do $900,000 of productive work. Those gains are rents, in the economic sense, not income. Giving those up--or downsizing, as you suggest--isn't ridiculous and is the point of the land value tax.
I would rather we deflate house prices by making it unprofitable to own a home that you don't live in, and have a massive oversupply of new homes.
Whenever I see inflation adjusted prices, I feel like I'm getting robbed.