Strongly agree. The biggest barrier between devs and refactors with zero interface impact is AGILE mentality of always aiming to deliver value to the end user with every deploy. Business people just don't understand how much of an impact crappy implementations and error handling does have on the end user. We see 5~6% error rate, which represents hundreds of thousand of people daily, but shipping a new "feature" (that within months will be used by a few hundred people with a very high bounce rate) is always the top priority.
From time to time there is a meeting with the following subject: "why are users not using the basic features", and the business people's answer is always "the ui is outdated" or "the menu is confusing the user". Those answers are gathered by talking to some random users on the streets close to the company building (I'm not making this up).