Pay off your mortgage. When you get pushed out of the workforce because of age (and you will), having no rent and no mortgage puts the situation in a different perspective. It becomes more of a beginning than an end.
People should always add up the total interest they're going to pay on a 30-year mortgage, some states require this to be prominently displayed on the loan. The amount is truly staggering.
Given that your average money-market is barely reaching 2%, the old saw of "it's better to save and service the mortgage" is complete BS and pro-bank propaganda. Mortgages are simply another wealth-transfer to the banks, including the mortgage-interest deduction: corporate welfare pure and simple. Read Liar's Poker if you want to really understand why that boondoggle is in place.
And never forget that easy mortgages, which have ludicrously enriched the banks, are the reason why real estate is so inhumanly overvalued now. The mortgage interest deduction was central to securitizing loans and we're all paying the price now.
2. You're concentrating all of your assets into a single object.
Instead of paying off your mortgage, you could put that money into other asset classes that grow faster, and you can distribute it to reduce your risk.
If you were laid off, you would have more money than if you had paid off your mortgage. Then if you wanted to get rid of your mortgage, you could and you would still have more money than if you had paid it off earlier. (Although even in that case, it's still generally better to leave the money invested, and just draw it down to pay the monthly payment.)
If you're really intent on exposing your savings to the real estate market, invest it in a REIT.. at least they can distribute your risk over a larger area and more buildings. And they'll invest it in markets that are growing faster than the average house.
Also, with main tax advantage I know of (interest deduction) you still pay more money if there is more interest.
i.e. if you have 20k of interest a year, the income tax deduction makes it seem like ~70% of that, depending on your tax bracket, or 14k. But if you're only paying 15k of interest per year, then it's 70% of 15k, or ~10k. 10k is less than 14k.
If they stay still or decline by all means, pay it down faster, but otherwise DO NOT pay off your mortgage early. Invest the extra cash in some cheap index fund instead.
If anything, in current markets more property is a better investment, and you can take advantage or gearing.