Hacker News Reader

Comment by simonw | Hacker News Reader

I got it working on my laptop! M2 64GB and needs 38GB of disk space:

  uv run --with 'numpy<2.0' --with mlx-vlm python \
    -m mlx_vlm.generate \
      --model mlx-community/QVQ-72B-Preview-4bit \
      --max-tokens 10000 \
      --temp 0.0 \
      --prompt "describe this" \
      --image pelicans-on-bicycles-veo2.jpg

Output of that command here: https://simonwillison.net/2024/Dec/24/qvq/#with-mlx-vlm

View on HN

Being able to run STOA models on my mac has been so cool. I am using it to let molmo vision via `mlx-vlm` click on my phone[1].

Although its 4bit quantised, it works surprisingly well.

1. Controlling phone using local LLMs - https://github.com/BandarLabs/clickclickclick

To be fair, if you pay Apple's prices for RAM upgrades then your computer better come with every bell and whistle in the book. The fact that Apple gets away with compute shaders instead of full-fat OpenCL compliance is a bit of a testament to how little they care about GPU compute.

+1 for also teaching me more cool uv flags.

Is the locally run model censored in the same way as the model hosted on HuggingFace? If so, I wonder how that censorship is baked into the weights—particularly the errors thrown when it starts to output specific strings.

OK, this surprised me:

  uv run --with 'numpy<2.0' --with mlx-vlm python \
    -m mlx_vlm.generate \
      --model mlx-community/QVQ-72B-Preview-4bit \
      --max-tokens 10000 \
      --temp 0.0 \
      --prompt "describe this" \
      --image Tank_Man_\(Tiananmen_Square_protester\).jpg

Here's the output: https://gist.github.com/simonw/e04e4fdade0c380ec5dd1e90fb5f3...

It included this bit:

> I remember seeing this image before; it's famous for capturing a moment of civil resistance. The person standing alone against the tanks symbolizes courage and defiance in the face of overwhelming power. It's a powerful visual statement about the human spirit and the desire for freedom or protest.

Thanks for testing that! (And, while I’m at it, thank you for your blog and for all of your contributions to HN as well. I am constantly learning from you.)

I wonder if the response was not truncated this time only because the word “Tiananmen” did not happen to appear in the response. At least some of the censorship of Chinese models seems to be triggered by specific strings in the output.

Both Qwen and QwQ are generally pretty easy to get talking about Tiananmen etc, so this isn't exactly new.

I wonder if it knows about Winnie the Pooh, or that character was wiped and don’t exist for it?

Damn, I'm struggling to run this on a M4 Pro with 48gb.

sudo sysctl iogpu.wired_limit_mb=45000

Reply on news.ycombinator.com