Hacker News Reader

Comment by treprinum | Hacker News Reader

Is there any tutorial on how to use HuggingFace LLaMA 2-derived models? They don't have checkpoint files of the original LLaMA and can't be used by the Meta's provided inference code, instead they use .bin files. I am only interested in Python code so no llama.cpp.

View on HN

I'd reconsider your rejection of llama.cpp if I were you. You can always call out to it from Python, but llama.cpp is by far the most active project in this space, and they've gotten the UX to the point where it's extremely simple to use.

This user on HuggingFace has all the models ready to go in GGML format and quantized at various sizes, which saves a lot of bandwidth:

https://huggingface.co/TheBloke

I understand, I use llama.cpp for my own personal stuff but can't override the policy on the project I want to plug it in, which is python-only.

There was a post yesterday about a 500 line single-file C implmenetation of llama2 with no dependencies. The llama2 architecture is hard coded. It shouldn't be too hard to port to python.

Found the repo, couldn't easily find the HN thread.

https://github.com/karpathy/llama2.c

That's Andrej's toy project, it won't run 7B LLaMA.

It does now: https://github.com/karpathy/llama2.c#metas-llama-2-models

>I am only interested in Python code so no llama.cpp.

llama cpp has python bindings: https://pypi.org/project/llama-cpp-python/

Here's using it with langchain: https://python.langchain.com/docs/integrations/llms/llamacpp

If you want high performance inference us this: https://vllm.ai/

For local batch stuff: https://huggingface.co/docs/transformers/main_classes/pipeli...

Reply on news.ycombinator.com