Tons of value left, especially for open source models. I would say the benchmark is yet to be truly saturated (just look at the legs and seat to see what I am talking about) and I always look forward to seeing them. Thank you!

To me the main upshot of this benchmark is precisely that the pelicans still usually look a bit wonky. It's bizarre, since this definitely has a good solution, but it's in line with my experience that memorization of the training set just... isn't happening very much? As in, whether a model fails or not doesn't have much to do with whether that exact question was likely posed many times before.

I think some additional value would be had by seeing how well it can modify the pelican.

Like, "now facing left", "sitting on the handlebars", or "with green spokes" to see if it can break out of some pretty obvious statistics in the training data!

And, there's always asking for an STL rather than an SVG!