2. You're concentrating all of your assets into a single object.

Instead of paying off your mortgage, you could put that money into other asset classes that grow faster, and you can distribute it to reduce your risk.

If you were laid off, you would have more money than if you had paid off your mortgage. Then if you wanted to get rid of your mortgage, you could and you would still have more money than if you had paid it off earlier. (Although even in that case, it's still generally better to leave the money invested, and just draw it down to pay the monthly payment.)

If you're really intent on exposing your savings to the real estate market, invest it in a REIT.. at least they can distribute your risk over a larger area and more buildings. And they'll invest it in markets that are growing faster than the average house.