Yeah, check out LLaVA-Plus (they call the experts in your vocabulary "tools") https://github.com/LLaVA-VL/LLaVA-Plus-Codebase