Your own argument is that you need a roof over your head, either way, so if the value doesn't increase, then you simply live there. If it does, then you keep all the profit, tax free.
Find yourself at a peak and you could be stuck for years to come.
Again, this was all about asymmetric upside. Being unable to move even as life circumstances change seems like a pretty sharp downside.