I always throw the Sally puzzle to any new model I try:

Sally, a girl, has 3 brothers. Each brother had 2 sisters. How many sisters does Sally have?

I’ve tried this on mistral, zephyr, llama variants. None of them get it right. Zephyr (on the HF demo page) shows me half a page of discussion and comes up with 8. Even gpt3.5 says 6, which is the most common answer among models.

Only GPT4 gets it right as far as I’ve seen.

I’ve heard a Mistral GPTQ variant gets it right but I haven’t found an easy way to run it.

If anyone found a local model that gets it right, please tell me exactly which one and how to run it!