I found progressive discovery works very well. I have an INDEX.md file in every folder with a table describing what files are in that folder and what they do.
Ah, so you are effectively offloading the file exploration mechanism to the INDEX.md in the sub-directories rather than writing a complex prompt?
Yes, exactly. I made the Context Layer into a skill where I describe the structure. So when I say, "reindex", the agent knows to check and update the index files. Also it knows if it modifies any file, it needs to also update the index. Works really well!