Usually the reason for me to leave a TODO is when something really minor doesn't work right and making it work would more often than not be a huge undertaking or at least require quite a bit of thought.
If the thing that didn't work right was important, I'd add a bug to the tracker in order not to forget it. If it was easily fixable, I'd fix it then and there.
This leaves these TODOs to be tasks I would not want a newcomer to the code base to tackle, because the first solution they would come up with is likely incorrect, so all the amount they'd put into fixing it might end up in vain.
That said: maybe others use TODOs differently. This really is judging from my own experience.