Hacker News Reader

Comment by faizshah | Hacker News Reader

If you make it a task prompt instead of a lateral thinking task you get a different “world model” or in my view you are just biasing the output to use a different structure.

```

Can you summarize joe’s schedule for my journal?

  Joe drove Sue to university. Afterwards he drove home again
    and drank a tea with her in the kitchen.

 Sure! Here is a summary of Joe's schedule for your journal:

 ---

 \*Joe's Schedule:\*

 - Drove Sue to university
 - Returned home
 - Drank tea with Sue in the kitchen

 ——-

```

View on HN

Me: Joe drove Sue to university. Afterwords he drove home again and drank a tea with her in the kitchen.

Explain.

ChatGPT: This statement seems to contain an ambiguity regarding Sue's location, suggesting some confusion about the timeline of events. Here are a few possibilities that could clarify the scenario:

1. *Sue Returns with Joe*: - Joe drove Sue to university. - Sue either decided not to stay at university or completed what she needed to do quickly. - Joe drove Sue back home. - They both drank tea together in the kitchen.

2. *Misstatement or Mistake*: - The statement might contain a mistake, implying that Joe and Sue met at home later but doesn't clearly explain how Sue got back from the university. - It could be intended to mean that Joe drove Sue to university and then drove home alone, and Sue somehow returned home later to join Joe for tea.

3. *Joe Visits University*: - Joe drove Sue to university. - Joe stayed at the university, possibly waiting for Sue to finish. - After she was done, they both drove back home together. - They then drank tea in the kitchen.

Based on the original statement, the first scenario seems most plausible, suggesting Sue quickly returned home with Joe.

Generally I think this would be a better test if the phrase were “dropped off” Sue at university. As phrased, it’s plausible to assume that Joe stayed at university with Sue, and drove her back and had tea after.

Though a truly smart model should seek to disambiguate situations like this.

I actually disagree with assuming an inconsistency when no time frame is specified and the phrase isn't "dropped off" as you suggested.

I tried giving Llama 3 70B a prompt that addresses both issues:

```

Can you summarize joe’s schedule for my journal?

4:00: Joe leaves the house with Sue 4:30: Joe drops Sue off at the university 5:00: Joe drinks tea with Sue in the kitchen

```

Reply:

```

I think there may be a problem here!

According to the schedule, Joe drops Sue off at the university at 4:30, but then he's supposed to drink tea with her in the kitchen at 5:00. That doesn't seem possible, since Sue is no longer with him after 4:30.

If you could clarify or correct the schedule, I'd be happy to help you summarize it for your journal!

```

Even Llama 8B can catch this once told `You think out loud before replying going forward`, which imo should be part of any system prompt for an LLM:

```

Let me think out loud for a moment...

Okay, so Joe's schedule starts at 4:00 PM, and he leaves the house with Sue. That's a good starting point.

Hmmm... then at 4:30 PM, Joe drops Sue off at the university. That means Sue is no longer with Joe at this point.

Wait a minute... but then at 5:00 PM, Joe is drinking tea with Sue in the kitchen. That's a bit confusing, since I thought Sue was already at the university.

Ah, I think I see what's going on here. It looks like there's a mistake in the schedule. Joe can't be drinking tea with Sue in the kitchen at 5:00 PM if he dropped her off at the university at 4:30 PM.

```

I even tried generating a synthetic schedule with a full day's worth of items: 70B still caught the contradiction, 8B no longer did

Reply on news.ycombinator.com