This is something people miss when talking about LLMs: „It’s the most average thing you could have possibly produced given the context.”

They can’t be creative by design. They’re useful when you want to reproduce, but not when you want to create something completely new (that you can maybe do by getting a bunch of average outputs from an LLM and getting inspired yourself).

When GPT-4 came out I was playing with it, and I often tried to get some unique, creative output from it, but very soon I learned it was futile. It was back when it all still felt magical, and I guess many of us tried various things with it.