Hacker News Reader

Comment by foota | Hacker News Reader

I'm not an ML expert, but I was thinking of a sort of "guided" embedding. E.g., give the image model some prompt for what it's trying to do? I don't understand why multimodal models generate an embedding that doesn't understand what the model is trying to "figure out".

I think this is similar to how Gemma 4 12B is implemented, but even then I don't think the single layer image embedding is "aware" of the context.

View on HN

You already do give the image model a prompt to tell it what to do. That’s not something the embedding can use independently of how the model is already using it.

In general an embedding doesn’t have intent or awareness in the way you’re looking for. “Embedding” just means one mathematical structure stuffed inside another. So for example the real number line is embedded into the Cartesian plane as each axis- that’s an embedding.

Now in this case specifically, the embeddings in any kind of transformer model encode the meaning of the thing they represent into vectors (which is what the model itself actually operates on). You can train the embedding to be more useful for a particular task at inference time, which already happens.

I think what I'm saying is that I don't understand why the embedding exists. I assume it's some kind of training and inference cost issue? But why can't the Gemma architecture linked above just learn to represent pixels in the LLM model's embedding space directly, rather than having the embedding from 48 x 48 pixel chunks? Or rather, give the embedding model some context to produce the embedding? (Which, as you note, wouldn't really be an embedding anymore, but seems like it would better understand fine detail)

Embeddings are used in language models to convert problems which are about words and meaning (which computers suck at) into problems which are about multiplying matrices, (which computers are really good at).

So embeddings are used in vision models to convert problems which are about the content and meaning of images into problems which are about multiplying matrices. The model doesn’t want to work with pixels (that’s what very basic vision models do, but it tends to be limited to special purpose applications) it wants to work with concepts in the image. That’s what the embedding gives it.

I still don’t really know what you mean about giving the embedding model some context. It embeds whatever you want to embed. So if you want to give it just a jpeg, fine. If you want to embed a jpeg and a json blob with some additional metadata/“context”/whatever, that’s also fine. That’s already how embeddings work.

The idea that I'm proposing basically isn't an embedding (which is context independent) but rather combining the embedding model with the context of the LLM. It sounds like the embedding model here is normally a "vision transformer" that maps image chunks into tokens with positional embeddings for both the position in the context as well as position within the image. Maybe it could be given the ability to consume the context and decide to emit multiple tokens for a single chunk? So for example, let's say that I ask a question "How many blades of grass are in this image" (a hard question for traditional image embeddings in models since the embedding won't contain the information). If the proposed architecture was both aware of the context and able to "decide" to emit multiple tokens for an image, then for the above it could emit tokens that represent the answer to the question posed, instead of just being the embedding of the image. Or you could ask something like "How many green pixels are there" and again I think it would work better under this architecture than it would otherwise.

I'm not sure how practical it is to train that architecture though or whether there would be performance issues.

Reply on news.ycombinator.com