Llama2-shepherd a CLI tool to install multiple implementations of the llama2

github.com

59 points·mikepapadim··8 comments

Are you planning on adding documentation / a mechanism for running a prompt using the code this installs?

As far as I can tell at the moment it clones one of the various repos for you and downloads some model writes, but it doesn't yet help you compile and run the code.

hello, I am planning to add also a runner option and a benchmarking option among different implementations. This is just an MVP version while trying to keep track of all llama2 implementations as the ones in the original repo is a bit outdated

It's sad that github has now defaults to showing a blank page devoid of any content related to the projects linked when js execution isn't complete.

Maybe Gothub can help with this: https://gothub.app/

It's like https://nitter.net/about but for Github. Just `s/github.com/gothub.app/g`

Do you have a performance comparison for inference on the same hardware using each of these implementations?

Not yet, this is the end-goal of this repo, to be able to do this kind of perf evaluation.

Everyone uses gguf / safetensors? How model management is done for this kind of tool?

most of the models support the tinyllamas, regarding gguf/ggml and safetensors each implementation has its own model importers, so there is not guarantee that all types can be consumed by all implementations

Reply on news.ycombinator.com