Hacker News Reader

I love Ollama's simplicity to download and consume different models with its REST API. I've never used it in a "production" environment, anyone knows how Ollama performs? or is it better to move to something like Vllm for that?

View on HN

The performance will probably be similar as long as you remember to tune the settings listed here: https://github.com/ollama/ollama/blob/main/docs/api.md

Try to, for example, set 'num_gpu' to 99 and 'use_mlock' to true.

They all probably already use elements of deep learning but are very likely trained in a supervised way to output structured data (I.e. actions)

+1

Reply on news.ycombinator.com