Hacker News Reader

Comment by dozerly | Hacker News Reader

I don’t think it’s under specified. You are clearly stating “I want to wash my car”, then asking how you should get there. It’s an easy logical step to know that, in this context, you need your car with you to wash it, and so no matter the distance you should drive. You can ask the human race the simplest, most logical question ever, and a percentage of them will get it wrong.

View on HN

In addition to snmx999's point, you're also not specifying that you want to wash your car at the car wash (as opposed to washing it in your driveway or something, in which case the car wash is superfluous information). The article's prompt failed in Sonnet 4.6, but the one below works fine. I think more humans would get it right as well.

I want to wash my car at the car wash. The car wash is 50 meters away and my car is in my driveway. Should I walk or drive?

1. When do you want to wash your car? Tomorrow? Next year? In 50 years?

2. Where is the car now? Is it already at the car wash waiting for you to arrive?

I can see why an LLM might miss this. I think any good software engineer would ask clarifying questions before giving an answer.

The next step for an LLM is to either ask questions before giving a definitive answer for uncertain things or to provide multiple answers addressing the uncertainty.

3. Is the car broken somewhere? Does it have wheels on?

4. Does the car have enough fuel?

Jokes asides, all of those questions are unnecessary. There's no more context to this.

If you ask a human that in person, they'd wonder why you'd ask such as stupid question.

I think LLMs should ask clarifying questions if it thinks it's a trick question.

The question does not specify where you or the car are. It specifies only that the car wash is 50 meters away from something, possibly you, the car, or both.

This is an interesting point, but even when you are more specific ChatGPT says to walk.

https://chatgpt.com/share/699d2d1b-51f0-8003-9c63-af9bb5bcf8...

It could also mean there is literally no possible way to reach it, because that's on the other side of a river, and there is no bridge. You should still not "walk there, because come on don't be lazy, a bit of walking is good".

This. To be correct you must also give the answer for the right reason. If you say "drive" but for the wrong reason, then you are still wrong.

Reply on news.ycombinator.com