Llamafile lets you distribute and run LLMs with a single file

All the new AI toys especially seem to love beefy newish hardware and especially GPU hardware if available

1 = ```

$ xcode-select --install

xcode-select: error: command line tools are already installed, use "Software Update" in System Settings to install updates

```

    $ ./llamafile-server-0.1-llava-v1.5-7b-q4
    [2]    25224 illegal hardware instruction  ./llamafile-server-0.1-llava-v1.5-7b-q4

    sudo wget -O /usr/bin/ape https://cosmo.zip/pub/cosmos/bin/ape-$(uname -m).elf
    sudo sh -c "echo ':APE:M::MZqFpD::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"
    sudo sh -c "echo ':APE-jart:M::jartsr::/usr/bin/ape:' >/proc/sys/fs/binfmt_misc/register"

See the Gotchas section of the README https://github.com/mozilla-Ocho/llamafile#gotchas

But yes, unless there is a way to patch it, bundling the model with the executable like this is going to be more wasteful.

    unzip -l llamafile-server-0.1-llava-v1.5-7b-q4 | grep llava-v1
    Archive:  llamafile-server-0.1-llava-v1.5-7b-q4
    4081004224  11-15-2023 22:13   llava-v1.5-7b-Q4_K.gguf
    177415936  11-15-2023 22:13   llava-v1.5-7b-mmproj-Q4_0.gguf

> llamafile-server-0.1 -m llama-2-13b.Q8_0.gguf

https://github.com/Mozilla-Ocho/llamafile/releases/tag/0.1

You know, most people don't have 24+GB GPUs sitting around to train these models. So in my book this is a huge step forward. Personally, this is the first time i am able to run an LLM on my computer, and it's purely thanks to this.

For GPT-4 the pricing is double that (0.09 USD per 1K). So only 200 larger interactions to reach 20 USD cost.

Or am I wrong?

Looks like cargo cult for me.

Ask it if is there life on Mars in that parallel reality

._.

Pain.

You could have multiple updates my using say iso time and doing a sort (so that ~/.freechat.run.20231127120000 would be overriden by ~/.freechat.run.20231129160000 without making the user delete anything)

    ./llamafile-server-0.1 -m llama-2-13b.Q8_0.gguf

My experience is that the world of Python dependency management is a mess which sometimes works, and sometimes forces you to spend hours-to-days searching for obscure error messages and trying maybe-fixes posted in Github issues for some other package, just in case it helps. This sometimes extends further - e.g. with hours-to-days spent trying to install just-the-right-version-of-CUDA on Linux...

Anyway, the (somewhat annoying but understandable) solution that some developers take is to make their utility/app/whatever as self-contained as possible with a fresh install of everything from Python downwards inside a venv - which results in (for example) multiple copies of PyTorch spread around your HDD. This is great for less technical users who just need a minimal-difficulty install (as IME it works maybe 80-90% of the time), good for people who don't want to spend their time debugging incompatibilities between different library versions, but frustrating for the more technically-inclined user.

This is just another approach to the same problem, which presumably also presents an even-lower level of work for the maintainers, since it avoids Python installs and packages altogether?

I think you’re imposing your own very specific (and solvable!) gripe onto an unrelated project and it makes for pretty clumsy commentary.

They made a PR that was built on top of another PR. The authorship information was preserved in the git history, and there was no attempt at deception. They also supposedly collaborated with the author of the original PR (which was never denied by either of them). All of this is totally normal working practice.

Those allegations of "stealing" just stem from a GH user piling onto the drama from the breaking change by pointing out where the initials from the new file format come from (which wasn't called into question on the original PR).

They were also not banned for those stealing allegations. They, as well as the author from the reversal PR were banned, as the maintainer deemed the resulting "drama" from the breaking changes to be a distraction to the project goals. The maintainer accepted the PR, and the nature of the breaking changes was obviously stated, so that drama wasn't completely on jart.

Mass adoption comes from ease-of-use, which includes ease-of-deployment.

Most of the HN crowd is savvy enough to run HF models directly in their terminal, but there's a reason Ollama, LM Studio, and Faraday are so popular.

Until LLM runners are part of the base OS, methods like this are extremely helpful to reduce friction.

About your question in the third paragraph: This is totally orthogonal to GGUF, and a cursory reading of the README shows that it does uses GGUF. This is not about a new universal LLM format, this is about packing it in a universal executable that runs everywhere, using Cosmopolitan.

Some examples do pack the executable and GGUF weights together in a single file, but that's not dissimilar from an self-executing zip, the only difference is that this executable is not OS-specific, so you can use the same exact binary for macOS or Linux, for example.

Well, it simplifies things when you don't need to compile things.

Also, you literally can't download or compile the wrong binary by mistake, it's the same binary for all supported processor/OSes Cartesian product matrix.

> Why introduce one more container?

It makes stuff more convenient.

`application/zip` is also a ubiquitous standard. I doubt anyone is being "introduced to it".

I also appreciate the fact that tooling for handling `application/zip` is very widespread, so you don't need totally bespoke tooling to retrieve the models from inside a `llamafile`.

> Who benefits from binary distribution of this sort?

Anyone that doesn't have a compiler SDK on their computer.

Thank you!

I assume the answers are "because Justine can" and "sometimes it's easier to distribute a single file than two".

From the user's point of view: they just want to run the thing, and as quickly as possible. If multiple programs want to use the GPU, then the OS and/or the driver should figure it out.

Assuming increasing vram leads to an appreciable improvement in model speed, it should default to using all but 10% of the vram of the largest GPU, or all but 1GB, whichever is less.

If I've got 8GB of vram, the software should figure out the right number of layers to offload and a sensible context size, to not exceed 7GB of vram.

(Although I realise the authors are just doing what llama.cpp does, so they didn't design it the way it is)

MLC LLM?

I think the binary it compiles down to (Probably the Vulkan and Metal ones for yall) is seperate from the weights, so you could ship a bunch in one file.

Seems like a dealbreaker for the whole idea.

    $ ./llava-v1.5-7b-q4-server.llamafile --help
    ./llava-v1.5-7b-q4-server.llamafile: line 60: /bin/mkdir: No such file or directory

yes, there are other apis (opengl,opencl) to access the gpu but they're all deprecated.

technically, yes, this is using Metal.

I'm guessing this should be 13.6?