They have no intelligence. These are very very very refined prediction engines.
> A common set is having Claude take some product photos and turn them into lifestyle ones using ChatGPT via Claude in Chrome. It's a pretty well-honed workflow at this point, but it'll still return images where the product is obviously not correct and seemingly not notice them.
Obvious to you or I, or someone with actual intelligence. But things like this slip by a frontier model in the same way AI from a few years ago would generate an image with seven fingers. They do not count. They do not understand. They do not consider.
No matter how good these models appear to be at intelligent tasks, it's foolish to give them "a company" to run, because they cannot understand when they've made a mistake the way even the least competent human can.