Ask HN: How do you work on things that will never make it into the sprint?

Say you're working on some functionality and notice a file that could be cleaned up and refactored, but it is unrelated to the current thing you're working on completely. It would probably take about a day or maybe even two to refactor it. You add it to the backlog and month after month passes and you never get to add it to the sprint. What do you do? Work outside the sprint? Never work on it?