In my last company, we were trying to ship refactors and improvements at the same time, and it inevitably broke down horribly when we'd discover some requirement that had originally never been documented or built in some component. So the scope for shipping would change from a one-to-one refactor with incremental iteration in subsequent sprints, to many cycles of QA and fixes before even minor design improvements could be shipped.
It's characteristic of every dev team I've been on that's tried to use off-the-shelf agile practices, because there's no risk tolerance at any level higher than IC. Everyone's afraid of how they'll appear, and nobody trusts anyone lower than them on the org-chart.