Hacker News Reader

Comment by iamniels | Hacker News Reader

I understand why you would like to use an LLM for vision. I do it myself often enough. I don't understand however, why the pill detection and counting is included in this benchmark. That is a task which you would perform with OpenCV right?

In my personal mini benchmark minicpm-v-4.6 scores amazingly well. Its a 0.8B model which runs fine on many consumer hardware.

View on HN

Generating datasets to train more efficient models is a common use case for VLMs, especially frontier ones. It makes it much cheaper to create that initial dataset and you can abuse the nondeterminism of LLMs to identify data for human review (if they don’t converge, escalate to a human).

Especially the pill counting example. The best model was shown at 81.1% accuracy, which is a terrible rate for pharmacy scenarios. It seems like implementors would be better off instructing the models to use deterministic tools (like OpenCV) until the models are at 99.99% accuracy (or whatever an acceptable error rate is for pharmacy techs).

I think that is because people perceive OpenCV as 'hard to use' and LLMs as easy to use.

OpenCV is no longer hard to use, it just takes longer. Still, a little more complicated than asking LLM to count.

To use an LLM, you just prompt it with an image + text saying "count the pills in this image".

To use OpenCV, ... you just prompt an LLM with an image + text saying "count the pills in this image, using OpenCV instead of eyeballing it".

(I like to throw in "produce intermediary artifacts so I can see the process" for more difficult tasks; this helps the model avoiding making hallucination-prone leaps and gives more opportunities to self-correct. At a cost of extra time and tokens, of course.)

Using OpenCV without an LLM? Nah, not touching that, I don't have free weekends to waste anymore.

I no longer use it but never felt it was particularly complicated, but since the days of resnet there are much faster ways to the goal.

Reply on news.ycombinator.com