This sounds similar to Microsoft's Autogen, and I think it's possible to replicate a lot of what you're talking about by using the rough structure of Autogen alongside the Assistants API
Maybe this would require additional research, but I think having a single GPT with access to all tools might be slower and less optimal, especially if the user knows exactly what they need for a given task and can reach for that quickly.