Comment by page_severian | Hacker News Reader

Comment by page_severian | Hacker News Reader

Parent Full thread

These errors are common and important to note. Though what I find fascinating is how often I can prompt engineer around these errors by telling GPT to just be smarter and think more carefully. Just tried the following for instance.

Q: You are a master logician, who can solve all manner of language problems and have excellent math skills. I have 2 cats and 1 horse. How many legs do they have in total? Show your reasoning and calculations step-by-step

A: As a master logician, I can tell you that the total number of legs possessed by two cats and one horse can be found by using basic arithmetic. Here's how:

First, we know that each cat has four legs, because all cats have four legs. This means that the two cats together have 2 x 4 = 8 legs.

Next, we know that a horse has four legs, because all horses have four legs. This means that the horse has an additional 4 legs.

Finally, we can add the number of legs possessed by the two cats and the horse together to find the total number of legs. This is done by adding 8 + 4 = 12 legs.

Therefore, the total number of legs possessed by two cats and one horse is 12 legs. I hope this helps! Let me know if you have any other questions.

Thanks for that. What a great example of how to actually use these things. You need to almost prime the chain with your prompt, give it some structure towards the particular later chain step you want. It makes a lot of sense that it can’t just leap to what you want, but with some setup, you can almost lay the path for it to follow.

That's interesting. It seems it's not great at raw calculation, but if you ask it to explain its reasoning, it derives steps from previous results, and arrives at the correct answer. It's similar to the sibling comment about asking it to tell a story.

Correct. That concept is the chain of thought (CoT) reasoning that the article discusses.

This seem to work because in the end it’s a markov chain. So the probability of the next step of a long answer being correct is greater than the probability of jumping to a correct conclusion.

Yes, this certainly helps. I find it ironic that you can get somewhat better results for nonfiction by giving it more clues about what it's supposed to be pretending to be. It's always pretending, though.

There’s an actual psychological effect for this as well [1]. Authors + uni are in the link. I forgot the name of the effect, don’t have time to do proper research.

[1] https://www.themarysue.com/lab-coats-increase-attention/#:~:....

0 chance this replicates, anything before 2016 I immediately write off. (i.e. clearly pre-replication crisis)

Halycon days of TED talks laundering cute little tidbits that seemed irrational but we all wanted to believe.

Fair point, not sure if it'll replicate. I just vaguely remember there's a thing in psych that if you act like it (a bit) then you become it a bit. Don't have the time to research it.

> Authors Hajo Adam and Adam Galinsky wanted to explore that nature of what they call “enclothed cognition,” the effect that the clothes we are wearing have on our psychology

Who isn't?

Suddenly, all those movies and tv shows where characters tell robots to concentrate start to make sense.

This looks similar to priming in humans.

Also, it looks to me like a language model is capable of reasoning if you let it execute a few times.

First, have it generate multiple outputs using different primings. Then have it choose it's favorite output.

Map, Reduce

Quite bizzare really. I wonder if you tweak a single word in that prompt, if you end up with a totally different sum.

Reply on news.ycombinator.com