I am also working on an open source CLI tool in this space [0]. I've taken a different approach, more focused on chatting with GPT to have it edit the code in your local git repo.
But my tool also provides GPT with a semantic map of your repo and the ability to ask to see particular files, etc. I use it to answer questions about unknown codebases all the time, and then start asking it to make changes. I have a chat transcript that illustrates that here [1]. As another example I needed a new feature in the glow tool and was able to make a PR [2] for it, even though I don't know anything about that codebase or even how to write golang.
Also, there's a small discord [3] where a few of us working on "AI coding tools" have been sharing ideas. You might be interested in joining the conversation over there.
[0] https://github.com/paul-gauthier/aider
[1] https://aider.chat/examples/2048-game.html