Hacker News Reader

Look into vllm and text-generation-inference

Why not use an API though?

Some to check out are:

Replicate, Together AI, Fireworks AI, Deepinfra, Anyscale Endpoints

I can personally recommend deepinfra. Been using it the past few days and it's served me very well.