Hacker News Reader

Comment by syntaxing | Hacker News Reader

TheBloke doesn’t joke around [1]. I’m guessing we’ll have the quantized ones by the end of the day. I’m super excited to use the 34B Python 4 bit quantized one that should just fit on a 3090.

[1] https://huggingface.co/TheBloke/CodeLlama-13B-Python-fp16

View on HN

Ollama supports it already:

`ollama run codellama:7b-instruct`

https://ollama.ai/blog/run-code-llama-locally

More models uploaded as we speak:

https://ollama.ai/library/codellama

Whoa, it’s absolutely astounding how fast the community is reacting to these model release!

while it supports it, so far I've only managed to get infinite streams of near nonsense from the ollama models (codellama:7b-q4_0 and codellama:latest)

my questions were asking how to construct an indexam for postgres in c, how to write an r-tree in javascript, and how to write a binary tree in javascript.

Similarly, I had it emit hundreds of blank lines before cancelling it.

What fortune, I so happen to need hundreds of blank lines.

Maybe it's outputting https://en.wikipedia.org/wiki/Whitespace_(programming_langua... :-)

Same, just tried it and it would give me infinite amount of blank lines

Sorry, this should be fixed now! To update you'll have to run:

  ollama pull codellama:7b-instruct

This is amazing!

I was up and running from clone/build-from-scratch/download in ~5m.

It's running on my M1.. it knows WebGL JS APIs better than I do, makes a passable attempt at VT100 ascii art, and well, should read more about Wolfram Automata, but does seem to know Game of Life!

Thank you so much!

still modifying the code completion (foundation / python models) to see what's causing the behavior.

Have had some good success with the instruct model:

codellama:7b-instruct

thanks! this give me some results, but I've had to use a specific construct to get anything meaningful:

using <language> write me a <thing>

it's managed to spit out code, rather than "write a traversal function".

Nice work with ollama!

Any plans to add the 13B quant models?

> managed to get infinite streams of near nonsense

This should be fixed now! To update you'll have to run:

  ollama pull codellama:7b-instruct

Isn't ollama terminal only? For code, that wouldn't be good.

They have a server/client model. The binary comes with a basic terminal front-end but you can just create your own self-hosted GUI or editor integration against the API[1]:

[1] https://github.com/jmorganca/ollama/blob/main/docs/api.md

Indeed! After pulling a model with "ollama pull codellama" you can access it via the REST API:

  curl -X POST http://localhost:11434/api/generate -d '{                        
    "model": "codellama",
    "prompt":"write a python script to add two numbers"
  }'

There is also ollama-ui. https://github.com/ollama-ui/ollama-ui

I'm testing it on my M2 Air (16GB). Quite fast!

What kind of cpu/gpu power do you need for quantization or these new gguf formats ?

I haven’t quantized these myself since TheBloke has been the main provider for all the quantized models. But when I did a 8 bit quantization to see how it compares to the transformers library load_in_8bit 4 months ago(?), it didn’t use my GPU but loaded each shard into the RAM during the conversion. I had an old 4C/8T CPU and the conversion took like 30 mins for a 13B.

i run llama2 13B models with 4-6 k-quantized oin a 3060 with 12Gb VRam

I can quantize models up to 70B just fine with around 40-50 GB of system RAM, using the GGMLv3 format.

GGUF seems not optimised yet, since quantizing with a newer version of llama.cpp supporting the format fails on the same hardware. I expect that to be fixed shortly.

For inference, I understand that the hardware requirements will be identical as before.

If I don't want to run this locally is it runnable somewhere on huggingface?

Replicate has already hosted Llama2 13B, the chat version. My guess is, in a short span of days or weeks they will host the code version too. They charge a dollar for 2000 generations if i am not mistaken.

https://replicate.com/a16z-infra/llama-2-13b-chat

can it be quantised further so it can run locally on a normal laptop of a developer?

“Normal laptop” is kind of hard to gauge but if you have a M series MacBook with 16GB+ RAM, you will be able to run 7B comfortably and 13B but stretching your RAM (cause of the unified RAM) at 4 bit quantization. These go all the way down to 2 bit but I personally I find the model noticeably deteriorate anything below 4 bit. You can see how much (V)RAM you need here [1].

[1] https://github.com/ggerganov/llama.cpp#quantization

Just started playing with this, there's a tool called ollama that runs Llama2 13B on my 16GB M1 Pro really smoothly with zero config.

Reply on news.ycombinator.com