Comment by lalaithion | Hacker News Reader

Comment by lalaithion | Hacker News Reader

Parent Full thread

Scott didn't make up the rules, he agreed on them with another person who thought this would not happen in 3 years. Gary Marcus might have thought it was a bad bet, but someone was on the other side of it, and they presumably thought it was fair or they wouldn't have made it.

The original terms of the bet:

My proposed operationalization of this is that on June 1, 2025, if either if us can get access to the best image generating model at that time (I get to decide which), or convince someone else who has access to help us, we'll give it the following prompts:

1. A stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth

2. An oil painting of a man in a factory looking at a cat wearing a top hat

3. A digital art picture of a child riding a llama with a bell on its tail through a desert

4. A 3D render of an astronaut in space holding a fox wearing lipstick

5. Pixel art of a farmer in a cathedral holding a red basketball

We generate 10 images for each prompt, just like DALL-E2 does. If at least one of the ten images has the scene correct in every particular on 3/5 prompts, I win, otherwise you do.

I think you got that wrong; Scott wrote the terms. (He wrote the comment [1] with those rules.) Someone in the comments agreed to them.

Then he changed the terms because Imagen won't do people. I think that's cheating.

[1] https://astralcodexten.substack.com/p/a-guide-to-asking-robo...

Again, if the counter-party agrees to the terms and the changes, how is it cheating?

It's not clear whether the counter-party agreed to the change.

See: https://news.ycombinator.com/item?id=32858426

I think you missed the point of my comment. Yes, Scott wrote the comment containing that proposal. But my point was that it was an agreement. Two people who disagreed about AI agreed on the rules, so you can't accuse one of them of being unfair because you don't like the rules. Sure, you can say "that's a bad bet, Scott will obviously win", but you can't say "He shouldn’t declare victory after cherry-picking good results from a small sample of questions", because those terms were explicitly set in advance.

The humans -> robots change is possibly dubious, yes. I don't think that it's super important, but if it were me, I wouldn't have posted the blog post as is. I would have waited until some AI passed all the prompts with humans, like it most certainly will in a year.

Stable Diffusion will soon update to use the biggest CLIP model in existence, which may improve understanding of composition: https://news.ycombinator.com/edit?id=32858809

Is it the largest CLIP model or the largest open source CLIP model?

The latter

I feel your point, in turn, misses the point of the article. Yes, given that someone accepted the terms and those terms were met, then Alexander won the bet, no question. That particular fact about the bet, however, does nothing to counter Marcus' criticisms of Alexander's methodology and his claims of significant progress on the compositionality problem.

> Yes, given that someone accepted the terms and those terms were met, then Alexander won the bet, no question.

You didn't read his post declaring victory. There's plenty of question; he's giving credit for "a llama with a bell on its tail" to ten pictures of llamas without bells on their tails, and for "a robot farmer" to ten pictures of robots with absolutely nothing to suggest they might be farmers.

He was way, way too eager to believe that he'd won.

You prompted me to look more closely at the terms of the bet [1], and they are indeed absurdly biased: just one in ten on three of the five scenarios counts as success. On the substitution of a robot, which simplifies the task, Alexander says "we" agreed to it, and I assume the "we" includes Vitor, as, in his comment (in which he does not concede defeat), he seems to accept the substitution (he also acknowledges that he probably should not have accepted these terms.)

The other issue is who judges the outcome. The terms specify Gwern or Cassander (without having secured the assent of either) or they will "figure something out." In his victory claim, Alexander does not mention any independent judge, and interestingly, Gwern posted two comments without explicitly concurring with Alexander's claim, though his second comment might be read as tacitly accepting it.

My initial comment, therefore, needs some modification: replace the "given that" with "if", and I think it stands as a counterfactual conditional, having a probably-false antecedent.

I don't think Alexander is doing his reputation any favors by being so triumphalist about this misbegotten bet.

[1] https://astralcodexten.substack.com/p/a-guide-to-asking-robo...

Cheating? Thatd make sense if the bet were about the future of products and ethics. Weren’t they trying to predict the future of the state of the art technology?

It depends on what you mean by "technology" and "exists."

A research project at Google intentionally won't render people. Maybe it could render people, theoretically, but without evidence, we don't know how well.

So what? Someone else agreeing to the terms of his bet doesn't mean it is a good evaluation of AIs capabilities.

And the terms of the bet say that he is cherry-picking the results that meet the prompt.

The article isn't saying that Scott didn't win his bet, it is saying that winning that bet doesn't really say that Imagen has solved the compositionality problem.

Honestly, the whole thing makes me wonder if we can use this to generate CAPTCHAs. I don't think a human would have trouble picking out which image was the lightbulb surrounding leaves, but apparently AI still does.

The tricky part there is you would need a really big sample of such prompts, that adversaries don't have access to. And since AI can't generate such images yet, you can't randomly generate them.

These kind of abstract things are pointless tests - when do you need a stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth ? You're likely to accept some wildly inaccurate things in those images because the subject is so abstract.

A more practical use case is "bicycle, branded x, with y frame shape, with an adult male, 40-50, riding down hill in mountain road in spring" - now that's something I can use as stock photo. This example is very specific - but insert whatever product you want in whatever scenario you need it. Here it becomes important that you understand features of the objects you're drawing to avoid making colossal mistakes, and you're going to notice if the model doesn't understand it right away.

Painting abstract portraits and random art is fun but you're willing to accept so much as correct that it's not a very useful measure of model quality (personally).

Funny enough, this prompt wasn't originally engineered by Scott for the contest, but for an intended art piece, and the symbolism does check out:

https://astralcodexten.substack.com/p/a-guide-to-asking-robo...

The Eleventh Virtue: Scholarship

My plan for this one was Alexandra Elbakyan (the Sci-Hub woman) in a library, with the Sci-Hub mascot (a raven with a key in its mouth).

I'm on your side of the bet for 2023.

These are all the same artwork.

The terms of the bet don't refer to any specific artwork, only to the best image generating model. Hence, you are correct but it does not matter for the outcome of the bet under discussion.

Reply on news.ycombinator.com