They respond so well that a lot of people believe their results without question, even while we know that they hallucinate all the time. They’re good enough for many. Even here in HN, where everybody should know that they are terrible with numbers, majority of us cheered that somebody generated diagrams with them, I couldn’t even find a single comment which were about “wait a minute, they’re fucking unreliable numbers”.

Also regarding coding, they can definitely produce junior level code, especially if you follow a similar trial and error way of thinking which happens all the time during your work, if you try to use an unknown interface.

Their ability to superficially mimic cognition, in a way that not only raises significant philosophical questions but also seems (rightly or wrongly) to bring the prospect of "true" AI tantalizingly closer.

A stochastic parrot, a blurry jpeg of the web is able to provide sound, actionable, verifiable advice on topics such as modeling in Blender or managing a Linux server -- despite not being generally intelligent.