Dodging an issue tracker is, IMO, one of the clear benefits of TODOs (or FIXME, HACK, BUG, and family). Especially in low-trust teams or whenever there's a need to avoid the sorts of politics that can arise between roles.
Refactoring tickets are terrible for all the reasons you point out. PMs will essentially never prioritize refactor work over new features unless pressured to, and if so, the work more often than not gets assigned to newer hires and junior devs, whereas a TODO is actually more likely to get picked up by a curious and intrepid newcomer when they're actually ready for it. Should said newcomer stumble across it before they are ready, it can also be a signal that they ought to reach out to a more experienced developer to ask some questions. For devs, refactor tickets may be useless overhead: in time spent writing, time spent defending, time spent estimating, etc. And in return, there's little glory or praise achieved in working on them; tracking too much time on refactor tickets may even attract negative attention from management. Then good luck if your QAs and agile leads are suspicious of any unit of work that a new test plan can't be written for.
A good TODO is sometimes just how the sausage is made.