Hacker News Reader

Comment by tellarin | Hacker News Reader

“ It actually knows what a giraffe is.” -> It maybe has a sense of what a giraffe looks like.

One of the interesting things about the "monkey touching iguana" photos is just how abstract that sense is.

Several of the pictures look like it's taken geometry from a 'touching' source and painted both 'monkey' and 'iguana' textures on to both figures, because on the one hand its model of relationships is too sophisticated to just copy/paste monkey and iguana photos from its library, and on the other hand it's not sophisticated enough to always treat "monkey touching iguana" as implying that the monkey and the iguana are discrete animals. (An interesting contrast with it being generally praised for being remarkably good at things like putting hats on animals' heads...)

On the other hand if you posted "monkey touching iguana" as a prompt on an internet forum - say a photoshop contest - you would get essentially all of these variants from actual humans as well when the task delved into being "meta".

People are certainly capable of wilfully misinterpreting the prompt because they consider monkey/iguana chimeras to be amusing to draw, or think that semi-abstract hybrid animals with a caption musing on the meaning of "touching" is a higher form art than mere representation. And in a photoshop competition they might all copy one silly interpretation because they think it's meme-worthy.

But I don't think if you asked 18 humans to independently draw "monkey touching iguana" you'd get 17 pairs of monkey/iguana hybrids mostly not touching each other against photographic backgrounds often featuring human limbs and one apparently normal monkey being pursued by a giant iguana!

I wonder what would happen if you put in the text prompt: "Long-necked spotted mammal that eats leaves from trees".

Most people would answer that as a giraffe.

I was curious about this as well. Here is what I got: https://i.imgur.com/Ey0i3My.png

Dall-E knows what a giraffe is IMO

I found it incredible too

The text model of dall-e at very least can map "Long-necked spotted mammal that eats leaves from trees" near the same representation of "giraffe"

Very similar to my images for the prompt: https://imgur.com/8dw2brf

Intriguing that it's gone for a headshot for all of them. I suspect it says something about the source text

I was curious about this too and wondered if it could then relate to my description of a giraffe so put in "legs of a yellow mammal with a long neck and spots" and it still nailed it.

The pictures are incredibly similar to what one gets when you type that same string in to google image search - it makes me wonder if you could do reverse-image search on some of the pictures it produces and find whole training pictures embedded in it, the way one can find blocks of intact training text stored inside language models.

Let's not play word games.

Do you know what a giraffe is? No, you just know what a giraffe looks like, where it lives, and maybe that it's vaguely related to a horse.

That last bit about the horse is pretty loaded. Knowing what a giraffe is means knowing that it is a mammal, has a blood system, eats leaves, reacts certain ways to certain stimuli, etc... That is a lot more than just knowing what it looks like. Image recognition is not the same a relationships or knowledge.

> mammal, has a blood system, eats leaves, reacts certain

DALL-E likely can map all those concepts to a giraffe also.

Now ask DALL-E "giraffe feeding young", "giraffe organs", "giraffe eating", etc, I think you'd find most knowledge is there.

What about ,"giraffe laying eggs"?

By your criteria, no human before the 18th century (when Carl Linnaeus invented the class “mammalia”) knew what a giraffe was.

And in the 25th century, if we manage keep technology alive, the dall-es of the day will be trained in more computer depictions of long extinct animals that accidentally made it into the training set than pictures of real animals. It will be Dürer's rhinoceros all over again.

That's not a word game at all. To know what a giraffe is is to be able to understand the implications of being informed that something is a giraffe. Dalle would never assume on its own that "a giraffe close to a low flying helicopter" could be very dangerous for the giraffe, but "a horse close to a low flying helicopter" is safe.

> Dalle would never assume on its own that "a giraffe close to a low flying helicopter" could be very dangerous for the giraffe, but "a horse close to a low flying helicopter" is safe.

Are you sure? (Can someone run down those prompts?)

Dall-e might be able to make those relationships in the latent space

> Are you sure?

Considering Dall-e has problems painting "a red cube on top of a blue cube" [1] and all kind of simple spatial relations, I'd say it's a fair shot.

[1] As reported by OpenAI, but there are also some prompts by Gary Marcus et al. (https://arxiv.org/abs/2204.13807) showing this, and it's trivially simple to find other very simple cases like these

You know a lot more than that. You know that it eats, sleeps, runs, mates. You know that there are male and female giraffes and what that means. You know that it dies if it gets too hot, too cold, if it falls off a cliff. You know that it has legs, a head, neck, etc and you probably know that it has a heart, brain, lungs, kidneys, etc inside.

You know, vaguely, how it interacts with other things and what it's likely to do around a fruit tree, or a lion, or fire.

Even if you've never been close to a giraffe, you can probably imagine what it looks like from close enough to see individual hairs in its fur.

A lot of knowledge is still missing from ML systems that don't interact with the world.

It's even deeper than that. Humans and giraffes are mammals, very close cousins in terms of evolutionary biology. The very nature of your experience of reality is essentially shared with the giraffe.

Part of our brains are lizard, both yours and the giraffes. Tech so ancient that it uses the same circuits and chemicals with crustaceans.

You can imagine what existence is like for a giraffe with pretty much 99% accuracy without consciously knowing a single thing about it.

A word-based image generator cannot.

What do you mean by "is"? I've spent a fair amount of time with giraffes in real life. Does that mean I know what one is?

I would really hope so. All of this conversation is devolving into a philosophical argument about 2 letter words. To know what something is relates to understanding some of its properties, and how it might interact with the world around it. Whether DALL-E knows what something is, is debatable. Whether you know what something is should be pretty clear.

I do not grok a giraffe in the same way I do the things I work on.

Not really close to horses at all. It's an even-toed ungulate, closer to deer, cows and sheep. Many, many tens of millions of years from horses.. the even-toed herbivores are closer to dolphins (!!) than they are to horses or rhinos.

Reply on news.ycombinator.com