Hacker News Reader

Comment by 331c8c71 | Hacker News Reader

Well, my experience was that ChatGPT produced utter nonsense including literally "1/19+1/19=2/19=1" when asked a simple probability question.

View on HN

Transcript of a conversation I just had with ChatGPT:

Prompt:

I'm talking with a user named "331c8c71" on Hacker News. According to statistics I found by Googling, the number of Hacker News readers living in the USA is 36%. Assuming this is correct, what is the probability that 331c8c71 does not reside in the USA? Please explain your reasoning.

Response:

If we assume that the statistic you found is accurate, then we know that 36% of Hacker News readers live in the USA. Therefore, the probability that any given Hacker News reader does not live in the USA is 100% - 36% = 64%.

However, we do not know if 331c8c71 is a typical Hacker News reader. They could be part of a demographic that is more or less likely to live in the USA. Without more information, we cannot say for certain what the probability is that 331c8c71 does not reside in the USA.

So, the answer is 64% assuming that 331c8c71 is a typical Hacker News reader, but it may not be an accurate estimate depending on additional information about the user.

---

Like I said, these models are far from 100% accurate, and there is lots they get wrong, but they clearly are capable of some kind of reasoning that goes beyond simple text substitution of training data.

The fact that just a few years ago even parsing the question would be an achievement. I feel like people are just glossing over how incredible this answer is.

Also it doesn't seem like such a leap to couple these language models with dedicated computation systems and similar. Think training a model to feed prompts to Wolfram Alpha to actually compute the results, then reporting back.

I feel like there is some contingent of people who are really bent on downplaying the achievements of AI as of late. Its objectively insane, yet somehow every discussion is still sprinkled with some form of "It told me 8x8=60 so I closed the window and never used it again"

I am impressed with LLMs but I think their inability to produce an honest "I don't know" instead of hallucinating is an issue.

True, but the problem is that they never know. They don't interact with the real world and have no way of verifying their training data's accuracy. Perhaps they could assign a confidence level to their response? But then, if they assign a response a high confidence level and produce an incorrect answer, it will compound their failure. Not only were they wrong, but they vouched for their wrong answer.

Google search was incredibly valuable immediately even if most links could have been rubbish. I can't say the same with the current LLMs

It is an incredible achievement that LLMs produce human-like output (e.g., wouldn't know if a gpt bot answers me unless we are discussing a topic where precision/accuracy are important) but they hallucinate (they are confident BS-generators).

The hype is that LLMs can solve any problem and replace humans (jobs). It is not so.

It may depend on what you do but I find it is easier/faster to do the work myself then to spot and fix [a possibly subtle] error in AI output. Though some of the specific things will improve in time and you can find tasks where AI is useful even today.

I don't see how the models can improve for general tasks (AGI) without being existential threat to humans (not just jobs).

"reasoning" is a strong word. "Pattern extraction and application" is a better description of what is happening.

In particular, LLMs fail miserably at tasks like "apply this simple pattern many times in succession" aka "for-loop", because they can't count in an abstract way, only on concrete contexts.

IQ test, the sat, and exact mathematics are very different fields though. ChatGPT etc. are very good at "emulating" by sheer force and size logic, exact math doesn't work with "emulation"

Reply on news.ycombinator.com