Hacker News Reader

Comment by fxj | Hacker News Reader

I am right now playing with it running it locally using ollama. It is a 19GB download and it runs nicely on a nvidia A100 GPU.

https://ollama.com/library/qwq

View on HN

Runs nicely on my AMD 7900XTX too.

How are AMD cards performing? I heard it was still very hit and miss in regards to support. Have you also tried things like F5-TTS?

7900XT here, previously 6900XT. Support for 6000 series and beyond is great on Linux. You have to use an override envar (Ollama has info on this in their readme). ComfyUI has instructions for anything based on torch.

TensorFlow is unusable.

Care to say how many tokens per second you're getting?

For the "How many r's in the word strawberry"

    total duration:       15.278476756s
    load duration:        14.982999ms
    prompt eval count:    47 token(s)
    prompt eval duration: 5ms
    prompt eval rate:     9400.00 tokens/s
    eval count:           377 token(s)
    eval duration:        15.257s
    eval rate:            24.71 tokens/s

15 seconds to count the number of r's in a string using 120 TFlops... Really puts into perspective that we've got a long way yet to go.

This specific model is designed to second-guess itself/ chack it's work. So the slowness is part of the point.

A100 geez. The privileged few.

Reply on news.ycombinator.com