This problem sounds like an excellent opportunity. We need a race to the bottom for hosting LLMs to democratize the tech and lower costs. I cheer on anyone who figures this out.
This is classic queuing theory, rate limits etc. I don't have an answer but I would look there.
What if you could group multiple of them. Long queries run on the group that’s commonly doing those. Shorter queries que faster because they’ll execute faster.