Hacker News Reader

Comment by belugacat | Hacker News Reader

Given that DALL-E is a giant matrix multiplication that correlates fuzzy concepts in text to fuzzy concepts in images, wouldn't one expect that there will be hotspots of nonsensical (to us) correlations, eg between "apoploe vesrreaitais" and "bird"? Intuitively feels like an aspect of the no free lunch theorem.

View on HN

Exactly this. At a high level, DALL-E is mapping text to a (continuous) matrix and then mapping that matrix to an image (another a matrix). All text inputs will map to _something_. DALL-E doesn't care if that mapping makes sense, it has been trained to produce high-quality outputs, not to ensure the validity of mappings.

None of this makes DALL-E any less impressive to me. High quality image generation is a truly amazing result. Results from foundational models (GPT-3, PaLM, DALL-E, etc) are so impressive that they're forcing us to reconsider the nature of intelligence and raise the bar. That's a sign of a job well done to me.

But if it's just mapping text to image then it would be fair to assume that using the same text would result in the same image. But does that actually happen?

No, it does not. It also doesn't always generate the same category of image. See https://twitter.com/realmeatyhuman/status/153173861680386457....

As much as people would like there to be, there really does not seem to be anything here. The original author doesn't think so, either (would need to refind the tweet).

I took the previous user's question as asking more like: Say there's an image in the training set with the description "Salvador Dali sitting at desk with pen and paper, 1957", and you put in the prompt "Salvador Dali sitting at desk with pen and paper, 1957", how close would the result be to that original training image?

It doesn’t have to be the same to achieve low loss. The date is not relevant, for example, for the image content. It can memorize a few images, but you can’t “compress” all of the internet’s images into text (if you can, make a startup with the most efficient compression algorithm ever made).

> it has been trained to produce high-quality outputs, not to ensure the validity of mappings.

Just nitpicking here a bit. It has been trained to ensure the validity of mappings, but only for mappings of valid prompts, where "valid" is vaguely "things that appear in the training set". On the other hand, it wasn't trained to ensure the validity of mappings of invalid prompts to images. It's an open question what that would even mean - someone in another thread here suggested it should output "not a valid prompt" in this case.

I have seen many bad abuses of the NFL theorem's name.

This is by far the worst.

It makes sense that it would have weird connections but the big claim here is that it outputs those connections as rendered text despite failing to output actual text is was trained on and prompted with. That sounds very unexpected to me and requiring a lot of evidence (that would be easy to cherry pick), though this debunking wasn’t convincing either.

Yeah. The problem here is that the network only has room for concepts, and hasn't been trained to see meaningless crap. Nor does it really have any way to respond with "This isn't a sentence I know", it just has to come up with an image that best matches whatever prompt it has been fed.

Reply on news.ycombinator.com