Hacker News Reader

Comment by jokethrowaway | Hacker News Reader

Cosmopolitan is a nice hack but I'd never use it.

I really don't see the point, cross-platform software with multiple binaries is absolutely fine. I've shipped cross-platform software and it's a one time trivial cost to prepare a multi OS pipeline. The actual problems come from dependencies and I'm sure you'll have to solve the same problems with cosmopolitan.

Llamafile is another hack I'd never use - why limit myself to the models supported by llamafile when I can just install llama.cpp? (the web interface is also part of llama.cpp https://github.com/ggerganov/llama.cpp/pull/1998)

Moreover, people with GPUs will likely want to use something with better performance than llama.cpp (gptq or exllama2)

View on HN

> I really don't see the point, cross-platform software with multiple binaries is absolutely fine. I've shipped cross-platform software and it's a one time trivial cost to prepare a multi OS pipeline.

That's like people who said "I could do this dropbox thing in a weekend with just rsync, ssh and ftp"

You think you can, but even if you might cover this "one time trivial cost" once, 1) it doesn't scale well so you'll end up not doing it that often 2) most other people can't.

> people with GPUs likely want to use something with better performance

End users like performance, but first they need to be able to use something at all.

If you start with "compile this binary then download this 4Gb file then run that command in this terminal" you've lost 99% of the potential audience

I think you are missing the forest for the tree if you only see a technological hack instead of the change of paradigm it allows.

I'm getting >50 tokens/second with an RTX 2070 Super 8GB on Mistral 7B with llamafile. More than adequate performance.

For context, I'm running this on my basic mainstream laptop (6-core, IGP, Ryzen 5 5500U) with 16GB RAM. I'm getting ~6 tokens/second for mistral-7b-v0.1.Q4_K_M.

> That's like people who said "I could do this dropbox thing in a weekend with just rsync, ssh and ftp"

And so begins the cycle of enshitification!

Actually if you poke around at it a little more closely you'll find there's also a version of Llamafile that just ships llama.cpp, enabling you to supply a model of your choosing.

Also the point of a tool like this to make the ecosystem more accessible to everyone, not just software developers.

gonna be awhile before sufficiently powerful hardware is GA for LLM

I mean, that depends somewhat on your use case. My primary development machine -- a Framework with a recent board and 32GB of RAM -- is perfectly capable of running smaller models on CPU, and while it's certainly easy to find where the ceiling is (and this obviously isn't the hardware profile of an average user), it's still more than sufficient for basic use

Even if it does also chew through my battery...

Reply on news.ycombinator.com