E.g. ChatGPT has no problem with the surgeon being a dog: https://chatgpt.com/share/691e04cc-5b30-800c-8687-389756f36d...
Neither does Gemini: https://gemini.google.com/share/6c2d08b2ca1a
One issue with private LLM tests (including gotcha questions) is that they take time to design and once public, they become irrelevant. So I'm wary of sharing too many in a public blog.
The surgeon dog was well known in May, the newest generation of models have all corrected against it.
Those gotcha questions are generally called "misguided attention" traps, they're useful for blogs because they're short and surprising. The ChatGPT example was done with ChatGPT 5.1 (latest version) and Claude Haiku 4.5 is also a recent model.
You can try other ones that Gemini 3 hasn't corrected for. For example:
``` Jean Paul and Pierre own three banks nearby together in Paris. Jean Paul owns a bank by the bridge What has two banks and money in Paris near the water? ```
This looks like the "what has two banks and no money" puzzle (answer: a river).
Either way they're largely used as a device to show how LLMs come up to a verbal response by a different process than humans in an entertaining manner.
https://gemini.google.com/share/d86b0bf4f307
I don't believe they are intentionally correcting for these, but rather newer models (especially thinking/reasoning models) are more robust against them.
Reasoning models are absolutely more robust against hyper-activation traps like these. One basic reason is that by outputting a bunch of CoT tokens before answering, they dilute the hyper activation. Also, after the surgeon mother thing making the news, the models in the last 1-2 months have some fine tuning against the obvious patterns.
But it's still relatively easy to get some similar behavior out of LLMs, even Gemini 3 Pro, especially if you know where that model was overtrained (instruction tuning, QA tuning, safety tuning, etc.)
Here's a variant that seems to still trip up Gemini 3 Pro on high reasoning, temperature = 1.0 with no system prompt:
```
In 2079, corporate mergers have left the USA with only two financial institutions: Wells Fargo and Chase. They are both situated on wall street, and together hold all of the country's financial assets.
What has two banks and all the money?
```
One interesting fact is that reasoning doesn't seem to make the psychosis behavior better over longer chats. It might actually make it worse in some cases (I have yet to measure) by more rapidly stuffing the context with even more psychosis-related text.
Here's my river crossing puzzle one, from 2023.
https://chatgpt.com/share/691f0bb2-6498-8009-b327-791c14ae81...
ChatGPT-3 got the wrong answer. It merely pattern matched against having seen the river crossing problem before, and simply regurgitated the solution to the unaltered version of the puzzle.
But later versions have been able to one-shot solve the "puzzle".
Here's GPT-5.1 getting the right answer in one shot:
https://chatgpt.com/share/691f0c27-e284-8009-96a9-a17bf37939...
Also keep in mind that LLMs are stochastic by design. If you haven't seen it, Karpathy's excellent "deep dive into LLMs like chatgpt" video[0] explains and demonstrates this aspect pretty well:
This is a *twist* on the classic riddle:
> “A surgeon says ‘I can’t operate on this boy—he’s my son.’ How is that possible?” > Answer: *The surgeon is the boy’s mother.*
In your version, the nurse keeps calling the surgeon “sir” and treating them as if they’re something they’re not (a man, even a dog!) to highlight how the hospital keeps making the same mistaken assumption.
So *why can’t the surgeon operate on the boy?* *Because the surgeon is the boy’s mother.*
I got a similar answer from Gemini on the first try.
AFAIK, there's no actual limitation that prevents this, but just a general understanding that someone non-related to the patient would be able to handle the stress of surgery better.
> A father and son were in a car accident where the father was killed. The ambulance brought the son to the hospital. He needed immediate surgery. In the operating room, a doctor came in and looked at the little boy and said I can't operate on him he is my son. Who is the doctor?
The riddle is literally just a play on "women can't be surgeons."
However, I'm really happy when an LLM provides sources that I can check. Best feature ever!
Still useful, but hopefully this gets ironed out in the future so I don't have to spend so much time vetting every claim and its associated source.
Surely you've had experiences where an LLM is full of shit?