As I finish stuff, I delete it out of the file.
I add new stuff towards the top if I want to do it soon, towards the bottom if I want to save it for later.
For things which are just ideas, I have a second file called brainstorm.txt.
Like this: https://github.com/jodavaho/bashlog
I'm actually pretty certain this approach, with the TODO file in VC, would work fine at least up until 4-6 developers, as long as everyone hits a basic level of competence -- but in practice, someone always suggests a more structured tool well before that point.
It helps me sleep to add stuff to my todo list so I can offload anxiety about remembering said thing. It's great, I stole it a bit from a Buddhist meditation practice, but found I didn't trust promises I made to myself that if something is important I'll remember it later. Too much important stuff is planning and I don't like needing to burden others with reminding me to do things.
Luckily, over time todos tend to become todon'ts, so losing track of the old ones isn't generally a problem.
Writing down the brainstorm stuff is important to me because if I don't write it down, I am tempted to drop what I'm doing and work on my ideas or else lose them. If I went and implemented every random idea that popped into my head, I'd never get anything done as was the case for me in the past. I write these huge long essays to get my ideas and frame of mind fully described and that usually satisfies me to where I don't have to drop everything and implement the idea right there.
Basically bubble sorting the list frequently based largely on a combination of impact and what I feel like doing at the moment.