Hacker News Reader

Comment by foota | Hacker News Reader

I think what I'm saying is that I don't understand why the embedding exists. I assume it's some kind of training and inference cost issue? But why can't the Gemma architecture linked above just learn to represent pixels in the LLM model's embedding space directly, rather than having the embedding from 48 x 48 pixel chunks? Or rather, give the embedding model some context to produce the embedding? (Which, as you note, wouldn't really be an embedding anymore, but seems like it would better understand fine detail)

View on HN

Embeddings are used in language models to convert problems which are about words and meaning (which computers suck at) into problems which are about multiplying matrices, (which computers are really good at).

So embeddings are used in vision models to convert problems which are about the content and meaning of images into problems which are about multiplying matrices. The model doesn’t want to work with pixels (that’s what very basic vision models do, but it tends to be limited to special purpose applications) it wants to work with concepts in the image. That’s what the embedding gives it.

I still don’t really know what you mean about giving the embedding model some context. It embeds whatever you want to embed. So if you want to give it just a jpeg, fine. If you want to embed a jpeg and a json blob with some additional metadata/“context”/whatever, that’s also fine. That’s already how embeddings work.

The idea that I'm proposing basically isn't an embedding (which is context independent) but rather combining the embedding model with the context of the LLM. It sounds like the embedding model here is normally a "vision transformer" that maps image chunks into tokens with positional embeddings for both the position in the context as well as position within the image. Maybe it could be given the ability to consume the context and decide to emit multiple tokens for a single chunk? So for example, let's say that I ask a question "How many blades of grass are in this image" (a hard question for traditional image embeddings in models since the embedding won't contain the information). If the proposed architecture was both aware of the context and able to "decide" to emit multiple tokens for an image, then for the above it could emit tokens that represent the answer to the question posed, instead of just being the embedding of the image. Or you could ask something like "How many green pixels are there" and again I think it would work better under this architecture than it would otherwise.

I'm not sure how practical it is to train that architecture though or whether there would be performance issues.

Reply on news.ycombinator.com