This is a big deal. Curious if vLLM or LMDeploy will use this new backend or not; I know LMDeploy currently has a custom Triton backend.

We've been struggling with various limitations of LMDeploy, vLLM, and ctranslate2 for a fine-tuned llama2 model, so I'm hopeful this is a much better solution.