Around here, all the doctors have big houses and nothing else. As soon as they pay off the house, they upgrade. The goal is to have the primary residence be the only significant asset. It's the one thing that can't be seized to satisfy a normal lawsuit judgement.
That's a huge assumption. There is implicit risk in debt: you might not be able to make payments due to you losing income in the future. If your income is gone, your investments might not be worth as much either, so those might not cover your debts.
There's also peace of mind. There's a psychological effect of having and paying debt.