Ollama has built in support [1] for gguf models on huggingface, and exposes a openai compatible http endpoint [2].

You can also just test it out using the cli:

ollama run hf.co/unsloth/SmolLM2-1.7B-Instruct-GGUF:F16

1. https://huggingface.co/docs/hub/ollama

2. https://github.com/ollama/ollama?tab=readme-ov-file#start-ol...