Unsurprising given how it works, but the higher-level implications are somewhat unexpected. These model are extremely good when they have creative freedom and are used to produce some kind of art. Poems, text in various styles, images without important details, and so on. But they fail miserably when you give them tasks that require world knowledge or precision. For example, when I asked a programming question I got a wrong answer. It looked just right on the surface, but the details were all wrong.

I say this is unexpected because a few years ago most people expected AI models to solve mundane tasks like automating Excel spreadsheet, while artists were considered pretty safe. Now the opposite has come to pass. The artistic power of these models is amazing, but I would never trust them to do anything that requires precision.

The biggest problem these models have is not necessarily that they produce "incorrect" (in quotes because they are not actually incorrect, just not what it's trained on) outputs, but rather that they have no built-in mechanism to express uncertainty in a meaningful way that's calibrated for the end-task of producing correct information.