In case you haven't seen this: https://x.com/JeffDean/status/2024525132266688757

We were promised full SVG zoos, Simon. I want to see SVG pangolins please

Larger models better understand and reproduce what's in their training set.

For example, I used to get verbatim quotes and answers from copyrighted works when I used GPT-3.5. That's what clued me in to the copyright problem. Whereas, the smallest models often produced nonsense about the same topics. Because small models often produce nonsense.

You might need to do a new test each time to avoid your old ones being scraped into the training sets. Maybe a new one for each model produced after your last one. Totally unrelated to the last one, too.

Because it is in their training set but it's unrealistic to expect a 2B or 4B model to be able to perfectly reproduce everything it's seen before.

The training no doubt contributed to their ability to (very) loosely approximate an SVG of pelican on a bicycle, though.

Frankly I'm impressed

because generating nice looking svg requires handling code, shapes, long context, reasoning and at 2b you most likely will break the syntax of the file 9 times out of 10 if you train for that. or you will need to go for simpler pelicans. might not be worth to ft on a 2b. but on their top tier open model it is definitly worth it. even not directly but just crawling a github would make it train on your pelicans.