However housing is so expensive out there that I'd have to either buy a much smaller house, or drain my savings and go into debt and work until I'm in my 70's to pay it off.

^^^ you've answered this yourself.