Hacker News Reader

Now I really want to see that diagram…

LLMs are bad at family relations. My test question used to be explaining Ahnentafel numbers (you are 1, any person's father is 2x, any person's mother is 2x+1), then explaining X-chromosome inheritance (men inherit only from their mothers, women in addition get their father's X chromosome unchanged), and ask them to list the Ahnentafel numbers of ancestors a man may have inherited X DNA from, in order, up to some generation.

(This is OEIS sequence A280873)

But it turns out that's overkill. Just giving them a single Ahnentafel number and asking if you could have inherited X DNA from them, trips them up.

But it turns out even that is overkill. Leave out X DNA entirely, and ask them to describe how Ahnentafel number 67 is related to you, and that's too recursive for them to figure it out.

Neat! As a human one can recognize that this question embeds a computation and use a standard trick - explicitly ask the LLM to use a program to generate the answer. LLM's are great at generating code but not necessarily that great at executing it "in head" (e.g., "what's the numeric integral of foo?" vs "write a Python program that computes foo"). Some instances of this are noticed by models themselves (I guess by now all know that they are bad calculators so would whip out code to do multiplication) but still a lot of them remain. Concretely, Claude 3.7 with "How is ahnentafel number 67 related to me? Use a program to help you." gets to "your father's father's father's father's mother's mother", whereas without the hint it indeed trips up in arithmetic and logic errors.

Reply on news.ycombinator.com