Hacker News Reader

Comment by stravant | Hacker News Reader

It's about precision.

Image models can be very off and still produce a satisfying result. Consider that I could literally vary all the pixels in an image randomly by 10% and you'd just see it as a bit low quality but otherwise perfectly cohesive image.

Language models have no such luck, the problem they're trying to solve is way "sharper", it's very easy for their results to be strictly wrong if they're off even a little bit.

So you need a much larger model to get a sufficient level of "sharpness" for text.

View on HN

Maybe another way to think of it is that the error correction part of image generation models is offloaded to the human visual cortex which is a very old evolutionary construct and thus had time to become very resilient? In case of text generation, maybe the error tolerance of the human brain is less developed as human-level language is a newer evolutionary invention.

It'd be interesting if the parameter/complexity requirements are actually similar once you examine the system as a whole, meaning machine _and_ human brain.

> image generation models is offloaded to the human visual cortex which is a very old evolutionary construct and thus had time to become very resilient

This is a very important point. A group of my colleagues (who are not tech people) are much more impressed with the image generation models than with the chat interface, even though the images are often whacky or just wrong. Yet the fact that it tried is impressive to them, with their minds managing to fill in the blanks.

I wonder how this compares to how a toddler speaks vs. paints/draws, which is typically better in the former than the latter. I'm both cases, we fill in the blanks in our minds.

Toddler speaking gets impressive/surprising quite fast, whereas the drawing usually does not. The most surprising thing about most toddler drawings is listening to the kid describe it or tell you about making it.

The consistency of descriptions is particularly surprising to me. Like you got a roughly circular collection of seemingly random scribbles, but they can tell you exactly which parts of it correspond to the person's nose, hair, arms, eyes, etc. And the descriptions seem to stay the same if you ask about the same picture on different days. Still not sure what to make of this phenomenon but it is fascinating.

I wonder whether video and metaverse generation models will be even smaller than an image model because of this mechanism.

The mapping and motion parts of the human brain are also old evolutionary constructs that could error correct the output of models.

It's kind of similar to audio vs video. Although audio requires less data and processing than video, it's much more difficult to get the audio good than the video and if the audio is bad or even missing for some time, it's useless, while if the video is bad, stuck or missing, it's not that big of a deal most of the time.

This is particularly true in a video conferencing situation. If the audio is bad, you miss out a lot. If the video is bad, it's not a big deal.

Deaf people would disagree :) if you talk in sign language on zoom missing video parts would ruin the conversation.

I don't think it's about precision, in the case of audio vs video - if you remove all the even columns from a video it would be similar to reducing quality, the same can be done with audio - removing half of the frequencies uniformly will just lower the quality.

That's a pretty specific case. You can get really good performance for a ton of tasks in video (video question answering, object identification and tracking, action recognition, etc) by just sampling a frame per second or even less frequently. Definitely can't do that with audio.

Right - The ‘palette’ for text generation is smaller: just 26 or so letters (plus some other characters), and if you put the wrong ones next to one another the result is garbage.

There’s something interesting in the fact that an image based system doesn’t need as much complexity to capture a semantic model as a verbal system does; I think there’s maybe a parallel there to the way that human minds find it easier to just ‘visualize’ some things as a basis for reasoning about them, but if we can’t ‘visualize’ and instead have to ‘think things through’ it’s a more intensive process.

Like, GPT has well known trouble counting - ask it for five things and it will give you four or six. Humans can offload some thinking about counting to visual/spatial reasoning though.

The pallette for LLM is tokens not characters.

Depends on the LLM. Character based LLMs exist, and even have advantages vs regular LLMs...

And if it is characters (as it is for some models), its more than 26 of them for English. Between space, case, punctuation, and digits, its basically 7-bit ASCII without most of the control characters (newline is semantically important, the rest not), almost 100 characters.

The number of tokens is more in the range of 40k to 250k tokens.

Right.

Whereas for pixels there are over 16 million.

How many more than 250k is that?

Over 16 million.

More like 256 of them, just an image is actually 3 layers with RGB. If you are going into combinations, I could easily say that triplets of tokens are actually what's important, and so 250k^3 is the real number of whatever.

Actually now that I think about it, neural networks work with real numbers, so a pixel is just 3 numbers. Typical input for an image model would then be around 300x300x3 values. While an input for a language model is around 2000 tokens, but while each token is inputted as an integer into the model, mathematically it represents a 250k length vector, so mathematically the input is 250k x 2000 values. So 90k vs 500M. Also pixels next to each other in an image are related, so you can reduce model size by taking advantage of that (CNNs).

I was having some gourmet crème brûlée with my friend Zoë at the café near the entrepôt in Åland's capital city and that made me realize it's but your naïveté when you say English is contained in mere 26 letters, for there is a soupçon of über important words that have diacritical marks in them.

> It's about precision.

This is utterly wrong. There is a huge amount of redundancy in images compared to language. This redundancy is why image models have yet to surpass language models. In some sense, language is much easier than the vision problem.

how do you measure "surpass"?

I think no one has bothered with using as many images as documents used to train GPT-3.5, to create as big of a model, and then RLHF as done to produce ChatGPT from GPT-3.5 is why image models haven’t surpassed language models.

At any level of scale of model and scale of training set, images models do surpass language models.

No way this is true. I have yet to see it. Image permutations are so easy to come by where as they are eventually infinite.

Reply on news.ycombinator.com