I get the desire to make self-contained things, but a binary that only runs one model with one set of weights seems awfully constricting to me.
I assume the answers are "because Justine can" and "sometimes it's easier to distribute a single file than two".
If the weights are 4GB, and the binary code needed to actually execute them is 4.5MB, then the size of the executable part is a rounding error - I don't see any reason NOT to bundle that with the model.
But, I think llama plus obese binaries must be targeting a very, very different community- one that doesn't build its own binaries, runs in any number of different locations, and focuses on getting the model to run with the least friction.
that assumption gets wrong very fast with nvme storage, even before you add herding effects
No, only as long as you compare against a very low number of machines with local nvme.
The sum of the bandwith available on typical storage device (even cheap and low end) will be at most times greater than what you have of your expansive top of the line cluster
If you have a single local storage, you don't have scale, so you won't have money for an expansive top of the line cluster either. But if you are wasting money on it, yes you will have more bandwidth, but that's a degenerate case.
If you have a few local storage machines, the assumption gets very wrong and very fast: 1 low end tier nvme=1 G/s at worst, one top of the line WD 990: 8G/s at best, so we're talking about a ratio of ~ 8 in the most favorable scenario.
Yes, the average user.
It is similar in my mind to the early days of Linux, where you had to compile it yourself and tweaked some compiler flags, compared to now, where most people don’t even think about the fact that their phone or Steam deck runs it.
That said, it does reduce the friction of getting an LLM up and running and the self-contained nature makes it sort of a dedicated program equivalent to Awk. It might open up possibilities, like the AI version of the Unix philosophy - do one thing well. A hyper optimized LLM could be used in pipeline of commands, for example.