Check out BentoML, which is the underlying serving framework used by OpenLLM, and it supports other type of models and modality such as images and videos.

From the description of the repo: "An open platform for operating large language models (LLMs) in production."