Dynamically caching and serving multiple LLMs for inference?

Greetings all! I am looking for a solution to dynamically serve and cache multiple LLMs for inference. I swear that I saw a solution for this previously and bookmarked it, but after searching for... probably over a week now through everything and combing the web again, I am coming up short.

I am currently using Ollama to serve multiple models, but it will unload and reload models as they are called and only keep a single model in memory at a time. I am currently working with a server that has 4xV100 32GB cards, and was hoping to serve embedding, chat, and completion models to fill up or more completely utilize that memory.

Thanks for any help or suggestions!