Hacker News Reader

Comment by embedding-shape | Hacker News Reader

At the bottom it says "Score 98.58", what measure is used for this score? It's kind of horrible, the perspective is all off (legs of the table makes that very obvious), the mouse/hamster has two mouths, a stub for a right paw, looks like left hand holds a melon on a stick or something, and there are pluses in the background for some reason. Not sure it'd call it "close to perfect" which the score seems to want to indicate.

View on HN

Do you prefer the fable one?

It's more "correct" but looks a lot worse in my opinion:

https://aibenchy.com/compare/openai-gpt-6-astra-high/google-...

No, both of them look awful. I genuinely am unsure if this is a meme you're making that's going over my head...?

Well, this was the state of SVG generation until a few months ago:

https://aibenchy.com/showcase/?page=2#showcase=67fc6d6c8e4c3...

https://aibenchy.com/showcase/?page=3#showcase=c215b5c915da6...

Ah, okay, that makes a lot more sense! Thanks!

Yeah, that's confusing, the score is for the entire benchmark, not for SVG generation only.

Good point about the mouths, I just noticed, lol

Imo, it's still better than most models, I personally like the stylized perspective.

You can view here all generations for all models: https://aibenchy.com/showcase/

And not to mention, the hamster is standing at the wrong end of the table.

Incredible, I've played table-tennis since I was like 14, and I didn't notice that egregious error yet I noticed all the other small ones! Thanks for pointing that out, should have been very obvious.

And also the hamster has two mouths

I think it might be standing at the wrong edge of the table too, worth adding.

I've replaced "Score" there with model ranking, to reduce confusion, thanks for the feedback!

Now I'm wondering why it's ranked #4 instead, not sure this reduced confusion :P

What panel of judges are you using for scoring/ranking this? Seems subjective enough to not be able to be ranked/scored at all

Yeah, ranking is for entire model, not only SVG generation. Maybe I remove it entirely and add generation date instead.

I was thinking to manually grade/rank the SVGs, but I decided against it, as it is indeed subjective.

I was thinking it could have at least a simple objective check (hamster doesn't have extra or missing parts, table has 2 sides, and net is in the middle, etc.).

Reply on news.ycombinator.com