You'll want to try llama.cpp [1]. The set of models that it can support is expanding [2]. Folks have also written services [3] that wrap around it.

[1] https://github.com/ggerganov/llama.cpp [2] https://huggingface.co/TheBloke [3] https://github.com/abetlen/llama-cpp-python