Hacker News Reader

Comment by zmmmmm | Hacker News Reader

Missing multimodal again?

It is so valuable in practise to be able to have the models see screenshots - I guess if they aren't in the benchmarks then nobody will focus on it. But it completely nixes these for some of my main use cases.

View on HN

Probably not what you're after, but I've considered having a separate small mm-model act as a seeing-eye dog for the bigger more capable one.

But how do you prompt this smaller model to give back information? I've tried that in the past but didn't go well. What I do is to send written handoff files between models to pass context around, but only had good results with big vision models as well.

So many possibilities for how you could glue it all together. However, when I send Gemma 4 12B in llama.cpp an image with no accompanying text, it assumes I want a description and gives me one.

I just tried with an audio file, and it transcribed the lyrics as I hoped. Then it made a bunch of suggestions about what do next, which I wasn't after. I could probably fix that by sending some text to narrow the scope.

I've tried this, too, and for whatever reason, simple things like having a subagent with vision capabilities hallucinates responses to the blind models. I don't know why, and it's not consistent, but I don't see this happening when the primary agent is a vision model.

I can’t come up with a use case where I couldn’t extract the image details using another, multimodal model and pass it into the GLM’s context with as many details as I need.

I think you lose a lot by not having the vision capability shared with the text. It is the joint reasoning across them where the power lies (the same model that sees the code and made the changes to produce the visual presentation, sees the image of it and reasons about it).

I mean this is assuming the thing you're working on has a UI? Not all of us work in that space.

I would assume that GLM 6 will be multimodal, but 5.x will be text-only.

Reply on news.ycombinator.com