On the contrary; since they are right next to the thing that needs work you always encounter them Just In Time. And you can just delete them really easily if you decide they're truly stale. This may not lend itself to planning or management visibility, but for a small private project I prefer them to the overhead of separate tracking.
For a small private project there's no organisation which doesn't work.
I don't have anything resembling an issue tracker, but I've begun collecting brain dump style notes in either a gist or the Apple notes app.
Some of my side projects have more process around them than my work projects because the side projects will definitely get shelved at some point and resumed much later. Future me is always happier when the process was applied.
That means they become scenery, not something in the foreground. They are lost.