Hacker News Reader

Comment by WJW | Hacker News Reader

What kills it for me (so far, perhaps they can fix it in the future) is that there is no way to know if you've actually asked ChatGPT correctly and it has given you a perfect document with the answer you seek, or if your question was slightly off and it has given a wrong answer, or even if your question was correct but it still confidently gives the wrong answer. It famously asserted numbers like 42 were prime early on, though that seems to have been fixed (although that may have been just through hardcoding).

View on HN

> It famously asserted numbers like 42 were prime early on

Oh it tried to convince me that four is not larger than one. :) It went like this:

At first I asked ChatGPT to summarise J.C. Owen’s contribution to geometric constraint solving. It gave me an answer which might be even correct. It sounded correct anyway. Then I have asked for a worked example of his method used on a toy geometric constraint system. It run into problems with that. Chiefly that the system is selected was under constrained.

When I asked about that it insisted that it is well constrained. It even volunteared the info that for a system to be well constrained it needs as many variables as constraint equations. Then it told me that the system it generated has 4 variables and 1 equation.

As a last question I asked if “Is four larger than 1?” and it resolved the seeming contradiction by telling me that in the context of geometric constraint solving four is not larger than one. One does learn a new thing every day. :)

In SQL, SELECT 'four' > 1; returns true as well.

Or you have asked correctly (or not, that's hard to tell from the onset) and has given you a non-factual answer. For example, I asked ChatGPT for what would be the best introductory book for a subject (added some more conditions to make it pretty clear-cut). It recommended me a book that sounded reasonable in title, with existing authors in the field… but the book didn't exist, at all.

Yes. It can even give name of research papers that never existed

Research papers that probably should exist given the graph of knowledge that it has. I spent about 15 minutes searching for a non-existent paper (I knew this was probably the case when I started, by choice, by choice) and it turns out the title was a mashup of two other papers in the domain.

Same error a human might make if they read too many papers too quickly. Or went to a film festival and was asked to describe the plot(s) of the movies they watched.

Or you could be getting Calvin’s dad kind of answers:

Calvin:how do they know the weight limit for a bridge?

Dad: they drive bigger and bigger trucks over it until it collapses, then they rebuild the bridge and put up the sign

To take your parenthetical further, I'm sure they'll get to that. It was only 5 years ago that searching Google for "how old is the Earth" said 6000 years because only the young Earth creationists put such a huge stake into SEO for their pages.

https://www.reddit.com/r/atheism/comments/70mdfj/recently_go...

Google or any search engine provides links to relevant documents, whereas ChatGPT functions as a pure summary. Neither the search engine nor language model summary should be treated as authoritative, but at least a search engine cites its sources.

I wish ChatGPT would give you the score(s) of the results it returns, so you could understand how correct it "thinks" the response is. I wonder if those confidently wrong answers are also low scoring.

you could probably ask it

I'd considered that -- in some cases it says it's absolutely confident, like with basic arithmetic, and in others it says:

> Without more information, I cannot estimate the accuracy or reliability of these claims. It is important to note that there may be conflicting or inaccurate information available online, and it is always a good idea to verify information from multiple sources before accepting it as fact.

which seems pretty reasonable.

You can ask it, and it may generate text with a score, but there is no particular reason that the score generated in the text would reflect any internal score it has.

I had a long requirements chat last night and did just that:

There are many self-hosted solutions that could potentially meet the requirements you have listed for your RSS aggregator and reader project. Here is a list of some popular options, ranked in order of their approximate percentage of compatibility with the features you have specified:

RSS Aggregator

| Percentage Score | URL |

| --- | --- |

| 90% | https://tt-rss.org/ |

| 85% | https://miniflux.app/ |

| 80% | https://newsboat.org/ |

| 75% | https://github.com/lobsters/lobsters |

| 70% | https://github.com/noahcoad/feeds |

| 65% | https://github.com/ssut/py-googlenews |

| 60% | https://git.gnome.org/browse/feedreader/ |

RSS Reader

| Percentage Score | URL |

| --- | --- |

| 90% | https://tt-rss.org/ |

| 85% | https://miniflux.app/ |

| 80% | https://newsboat.org/ |

| 75% | https://github.com/lobsters/lobsters |

| 70% | https://github.com/noahcoad/feeds |

| 65% | https://github.com/ssut/py-googlenews |

| 60% | https://git.gnome.org/browse/feedreader/ |

Please note that these percentages are approximate and are intended only as a rough guide. You may want to consider other factors such as the user interface, performance, security, and integration with other tools when choosing a solution for your project.

These scores would mean nothing! At best it would score "how well does my answer fit to my model of the English language". At no point can the current architecture accurately grade "how correct is this answer in the context of the subject domain".

The GPT-3 playground lets you view the probabilities of generated tokens. That's the closest thing to a confidence score you can get from a transformer model. But like the other person said, this isn't very meaningful.

It tried to convince me that one event can happen before another event even if the time of the event is after the time of the other event. It had a notion that events happen in some order, and that time is ordered, and that events are associated with times, but not that the two orders need to be consistent. And I wasn’t able to convince it otherwise.

The way you describe your interaction is fundamentally flawed.

At no point was it trying to convince you of something.

At no point does it have any clue that "events occur in some order". Or that "time is ordered"...

At no point will you EVER be able to "convince it otherwise".

This is a language model. All it knows is that "in my training set, something said something akin to the two English words: 'ordered events'". It has no model of facts, deduction, or logic, as long as its results are a reasonable approximation of the English language as influenced by your prompt.

Yes, that‘s the conclusion I drew after a number of interactions. “It tried to convince me” was shorthand for that it persistently argued for that. I don’t attribute any actual agency to ChatGPT, of course.

The lack of self-reflection, of logical reasoning ability and associated memory is what makes ChatGPT useless for the use cases I’d actually find interesting.

1. Does Google inform you that you have asked a bad or unproductive question? 2. Does Google guarantee the links it provides you with contain high quality and relevant information? 3. Does Google ensure that the #1 link contains the correct answer?

I think we are forgetting that Google suffers from these problems too. A human who asks bad questions will continue to ask bad questions. AI should in general be trained to do what the human wants. If the human wants something stupid, that's on the human. Google also isn't guaranteed to find the right information you're looking for, especially if it is on an obscure topic or if the query is highly overloaded or being taken over by current trends.

The crucial difference is that the Google search results page, by definition of it's function, cites every source.

GPT vomits up word soup. How should you expect to verify every factoid contained? As others have pointed out: if you ask GPT to cite its sources it will very happily, and authoritatively, spit back articles and authors that do not exist.

This is a problem now, but I'm getting tired of these articles seeing a current limitation and acting as if that limitation will be the case forever. They'll iron out some wrinkles over time.

Reply on news.ycombinator.com