Hacker News Reader

Comment by behnamoh | Hacker News Reader

[flagged]

The ollama application has zero value; it’s just an easy to use front end to their model hosting which is both what this is and why they’re important.

Only having one model host (hugging face) is bad for obvious reasons (and good in others, yes, but still)

Ollama offering an alternative as a model host seems quite reasonable and quite well implemented.

The frontend really is nothing; it’s just llama.cpp in a go wrapper. It has no value and it’s not really interesting, it’s simple stable technology that is perfectly fine to rely on and be totally unexcited or interested in, technically.

…but, they do a lot more than that; and I think it’s a little unfair to imply that trivial piece of their stack is all they do.

The software that controls the front-end has enormous value, it becomes the central point and brand to manage and self-host LLMs that's used to manage 100 GB catalog of models which acts like a moat inhibiting switching to alternatives. Awareness and user-base are the hardest things to obtain with new Software products and it has both - right now it doesn't look it's monetizing its user base, but it could easily attract millions in VC funding to spin off a company to sell support contracts and "higher value" SaaS hosting or enterprise management features.

Whilst it's now a UX friendly front-end for llama.cpp, it's also working on adding support for other backends like MLX [1].

[1] https://github.com/ollama/ollama/issues/1730

I’m not seeing what the issue is with ollama. Can you elaborate? There are tons of open source projects that other stuff gets built upon: that’s part of the point of open source.

I might be wrong about this but doesn't ollama do some work to ensure the model runs efficiently given your hardware? Like choosing between how much gpu memory to consume so you don't oom. Does llama.cpp do that for you with zero config?

Yes, Ollama automatically determines the number of layers to offload based on available VRAM.

I would even say that Ollama is a step back. For example llama.cpp supports vulkan, which is a huge gamechanger for consumer grade hardware. Ollama does not support vulkan, eventhough it's probably fairly easy to do so.

If you care about running efficiently on your hardware, then llama.cpp is they way to go, not ollama.

I thought ollama was just a convenience wrapper around llama.cpp?

That might be how it started, but there are differences. For example support for LLama 3.2 Vision was added to Ollama[1], but not upstreamed[2] to llama.cpp due to image processing requirements AFIAK.

[1]: https://github.com/ollama/ollama/releases/tag/v0.4.0

[2]: https://github.com/ggerganov/llama.cpp/issues/9643

Looks like you’re being downvoted. It’d be nice if somebody could explain the difference, cause I’m also kinda out of the loop on this

It's very hard to put into words without coming off as being unfair to one side or the other, but the ollama project really does provide little-to-no _innovative_ value over simply running components of llama.cpp directly from the command line. 100% of the heavy lifting (from an LLM perspective) is in the llama.cpp codebase. The ollama parts are all simple, well understood, commodity components that most any developer could have produced.

Now, applications like ollama obviously need to exist, as not everyone can run CLI utilities, let alone clone a git repo and compile themselves. Easy to use GUIs are essential for the adoption of new tech (much like how there are many apps that wrap ffmpeg and are mostly UI).

However, if ollama are mostly doing commodity GUI things over a fully fleshed-out, _unique_ codebase to which their very existence is owed, they should do everything in their power to point that out. I'm sure they're legally within their rights because of the licensing, but just from an ethical perspective.

I think there is a lot of ill-will towards ollama in some hard-core OG LLM communities because ollama appears to be attempting to capture the value that ggerganov has provided to the world in this tool without adequate attribution (although there is a small footnote, iirc). Basically, the debt that ollama owes to llama.cpp is so immense that they need to do a much better job recognizing it imo.

Does Ollama offer a GUI? I don't think they do.

I use them because they run as a systemd service with a convenient HTTP API. That's been extremely helpful for switching between GUIs.

I also like their model orgazation scheme, and the modelfile paradigm. It's also really handy that it loads and unloads models when called, which is handy for experimentation and some complex workflows eg embedding followed by inference.

Is llama.cpp doing 100% of the "heavy lifting"? Sure, but some light lifting is also needed to lower the activation threshold and bring the value to life.

I would not use llama.cpp, it's simply too cumbersome.

If Ollama did not exist, I would have to invent it.

Is it not "innovative"? Who cares! I want it. Commodity GUI? Again, I don't think they have a GUI at all. Are you maybe thinking of OpenWebUI?

I think we agree on almost all points, but I thought ollama-gui was an official gui, so I’m even more baffled as to what the draw is. Running as a llama.cpp as a service/API endpoint is trivial (I do just that). Maybe you can outline for me what the value proposition of ollama is so I can better understand what it does that plain llama.cpp doesn’t.

Ollama allows me to use a single podman command, which uses the latest version of ollama, downloads a model of my choosing, and starts a local http endpoint widely supported by different clients. I can just run this one command to chat with a local model through a web interface, get code completions in VSCode, ask about the content of my local Markdown notes.

Now, I don't use AI that much, I could totally live without this. But if it weren't for the robust one-liner I probably wouldn't use local LLMs at all.

My experience of ollama is that it makes it super easy to pull various models and use them locally. Sure, I could do this myself but it's helpful not to have to.

I'm not going to pretend to know everything about Ollama, llama.cpp or llamafile, but my experiences using llama.cpp and llamafile (llama.cpp based) were both negative. Web UI frontends aren't relevant here, this is just purely about whether I can load a model and get it to produce coherent results that are in the realm of what the people who created the model intended.

With llama.cpp or llamafile, I was constantly having to look up a model's paper, documentation or other pages to see what the recommended parameters were, recommended templates were, and so on. My understanding is that GGUFs were supposed to solve that, yet still I was getting poor results.

You know, I don't know all the details or if there's any difference between what Modelfiles are for versus what GGUF metadata is for, but my experiences with Ollama have been that it just worked. It took me a while to even try Ollama, because the expectation is that it would simply be another interface on top of the same issues.

There are things I don't like about Ollama, but mostly they were easy to work around by writing a few scripts. Not using any web UI with it at all.

Reply on news.ycombinator.com