Hacker News Reader

Comment by wildrhythms | Hacker News Reader

I like to try giving logical puzzles to these chat bots to see what they say. I don't really expect them to do the reasoning to come to an answer, but it's interesting to see that they just make up new rules to satisfy some answer it thinks I want:

Prompt: Sally will go to the dance if James goes to the dance. Michael will go to the dance if Julie will go to the dance. James is sick and can't go to the dance. Will Michael go to the dance?

>No, Michael will not go to the dance. If James is sick and can't go, then Sally won't go either. And if Sally won't go, then Julie won't go, which means that the condition for Michael to go to the dance (that Julie will go) is not satisfied.

I think 'actual reasoning' would mean ChatGPT can solve (albeit contrived) puzzles like this. It can't, at least not reliably from my poking and prodding.

View on HN

When prompted with:

> Sally will go to the dance if James goes to the dance. Michael will not go to the dance if Julie will go to the dance. James is sick and can't go to the dance. Julie and James always go to dances together. Will Michael go to the dance?

ChatGPT responds with:

> Michael will not go to the dance, since his attendance is contingent on Julie not going and James being unable to attend due to illness does not change that condition. Given that James and Julie always attend dances together, it is likely that Julie will also not go to the dance, meaning that Michael's condition for attending will be met, and he will not go to the dance.

It's really good at fibbing, but not so good at 'actual reasoning' (whatever that may be).

I got:

> We know that Sally will go to the dance if James goes to the dance, and James is sick and can't go to the dance. Therefore, we can conclude that Sally will not go to the dance.

> Since Michael will go to the dance if Julie will go to the dance, we don't have enough information about Julie's attendance to the dance. We only know that James is sick and can't go to the dance, so it's possible that Julie could still go to the dance. If Julie does go to the dance, then Michael will go as well. However, if Julie doesn't go, then Michael may or may not go to the dance, depending on his personal decision.

That strikes me as more nuanced than either of the other two. Worth mentioning I'm paying for the Plus subscription, though, and we just got a new "Turbo" model that answers faster. I think that model may be allowed a little more power as well, so the answer quality might be slightly better.

my 8 year old would fail at your puzzle. is an 8 year old capable of "actual reasoning"?

this[1] was an interesting read. Particularly the 'Emergent Prompting Strategies' and 'chain-of-thought prompting'.

I think we're a long way from sentient AI, but there is a real sense of "something" unusual and heretofore not achieved in computing. The responses to logic queries are a long way ahead of statistically driven word mashing.

[1] https://ai.googleblog.com/2022/11/characterizing-emergent-ph...

Ask ChatGPT to turn math word problems into executable JavaScript. Then evaluate the JavaScript. Suddenly ChatGPT is much better at math.

But that only proves the limitations are there.

Yes, LLMs are different. They are not reliable computers and computers are not reliable translators.

Perhaps pick the best tool for the job?

Or just flail around wildly and leave snarky comments on articles about bullshit generators. Your choice.

Reply on news.ycombinator.com