Hacker News Reader

Comment by lopkeny12ko | Hacker News Reader

Because it's impossible to get started as someone who has no background in ML. I tried earlier this year and also failed.

1. Obtain the Llama models. Apparently you have to sign up for access and get a download link? I don't want to do that, found some public download links instead. Ok, now I have a few hundred GBs of model files.

2. Compile llama.cpp. Missing dependencies, took an hour to figure out how to resolve.

3. Quantize the model? What does that even mean?

4. Install pytorch. Run the command in the README. Python exception.

5. Install NVIDIA helper libraries. Doesn't work. Try installing the AMD helpers to run on CPU instead. No instructions for how to do this. Eventually figured it out.

6. Try running pytorch again. Same exception.

I gave up after a full day of trying to make this work. The Docker image is for people like me.

View on HN

Yeah it's great, you can get this running in no time (even without gpu, just to play with it):

  docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
  docker exec -it ollama ollama run llama2

And that's all!

It's nice to have this possibility on Linux now. It wasn't much more complicated on macOS before:

    brew install ollama
    brew services start ollama
    ollama run llama2

Works great on my M1 MacBook Air, although it's definitely not as good as ChatGPT.

And with GPU just do this for the first command

    docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

You'll need to have the NVidia container toolkit installed.

Thank you!

For docker-compose, do

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Docs: https://docs.docker.com/compose/gpu-support/

Btw, the quantized models are on Huggingface, so step 1 and steps 3-6 can be avoided. (The link is 3/4ths into the llama.cpp README)

Oh no, gotta RTFM

Why is everyone so afraid to get their hands dirty?

This isn't really fair. There are many manuals with rapidly changing or incomplete information, or information only months old that is no longer accurate.

This is a fast moving space and can definitely be confusing.

Regardless, people can do their research. They just don't. Yes this is a fast moving space, personally it's exhausting to keep up on.

But rather than pushing people to improve themselves or LEARN we cater to spoiled lazy child syndrome.

Put in the work, reap the rewards. No shortcuts!

I agree in principle, however you appear to be arguing in bad faith.

The person you responded outlined their process (to a level you can look at and see they clearly made an effort) and stated they put a full day in. This is not "spoiled lazy child syndrome", although there are for sure those out there who fit that mold.

got it thanks for the explanation!

helps the command line actually work

Reply on news.ycombinator.com