Hacker News Reader

Comment by davibu | Hacker News Reader

The criticisms regarding chatGPT remind me of what was said about Wikipedia at its very beginning, that it was supposedly unreliable. I think we will have a good laugh in a few years reading these first comments.

There is no doubt that chatGPT is the future. It is certainly perfectible, but the existing basis is a revolution in progress.

In my opinion, there are two essential things missing for chatGPT to become the perfect replacement for Wikipedia and Google: - The ability to activate a "system 2" or slow thinking (theorized by Daniel Kahneman) - The ability to cite sources

And the cherry on the cake would be the ability to interact with images

View on HN

I think the BS-generation problem with ChatGPT goes far deeper than citing sources, for a variety of reasons.

1) It's not a search engine, even if it behaves a bit like one. It's not "retrieving answers" to your questions (from sources that it could choose to cite). ChatGPT is really just a "language model", so it has no notion that what you're typing is even a question/query .. your input is just treated as sequence of words (which ChatGPT has zero understanding of), with ChatGPT's response then being a further sequence of words that it has calculated are (one) statistically probable continuation of what you typed (you can keep asking it for alternative answers, and it'll continue generating additional alternative statistically probable continuations).

The websites/etc that ChatGPT was trained on are just sources of language that it consumed in order to learn the statistics that let it make these continuation predictions. It's not memorizing "facts" from websites, just word statistics, and these are mixed in with the statistics from all the other sources it was trained on. If it generates the word "walk" as part of a response, it can't cite a source for that since there essentially is none - only a bazillion text sources it was trained on that collectively made the word "walk" a high probability continuation on the words it had generated leading up to that...

2) Even if ChatGPT had been designed to deal in "facts" (rather that words statistics) associated with specific sources, the bullshit problem isn't just knowing the varied reliability of the sources it was trained on, but how those "facts" are combined. To combine multiple facts and correctly deduce something new from them would require intelligence, but ChatGPT doesn't have any intelligence - it's just a statistical word generator, so the way it combines snippets from different sources is again just statistical word generation, with zero knowledge of the meaning of the words it is generating or whether it makes sense!

What makes ChatGPT seem semi-intelligent is that a lot of what it was trained on was text written by semi-intelligent humans, so the "sequence of words" it is generating, following the statistics of human speech, seems like something a human might say... until you start paying attention to the meaning of the words and realize it's often good-sounding garbage.

the thing is.. ChatGPT doesn't have to compete with perfectly correct information because the information you search for on Google is often wrong(=SEO spam) too and you have to sift through a lot of garbage or misleading links there too. Sometimes literally, because you get a forum link with a bunch of people saying wrong things and then finally someone says the answer. That's similar to what you have to do on ChatGPT to doublecheck or ask a follow up questions or read more or treat a piece of information with a dose of healthy doubt. Both ChatGPT/Google are very useful and they both produce imperfect results and they both require some human thought.

interesting point about SEO. Does ChatGPT somehow filter out SEO content? Is it rather selective about the domains it crawls? Because Google could certainly turn that switch too - but then it would lose its comprehensiveness ...

Which is the big problem. ChatGPT will produce something reasonable if it's seen good content on the indicated subject. Otherwise, it just makes up plausible blithering, including fake references.

Useful for fiction, advertising copy, and literary criticism. Not so good for fact retrieval.

If you go to the OpenAI playground and turn down 'temperature' to 0 it seems to not BS at all as far as I can tell.

Temperature is presumably referring to sampling the output probabilities. With a temperature of 0 it'll be giving you the very highest probability continuation, while with increasingly higher temperatures it'll be sampling from the possible continuations to provide more variety.

In other words, the temperature is controlling the variety of output, but of course doesn't affect what was fed into it in the first place. As the saying goes, Garbage-In, Garbage-Out .. even with a temperature of zero it's still going to be bullshitting since "predict next word" (language model) is fundamentally a bullshitting technology - just keep on spewing out words regardless of meaning.

I put in a Python script I wrote to automate some things around AWS. It described the purpose of the scripts. Then I asked it make some changes and it did. I asked it why would I use it. It gave me a plausible explanation. I asked it to add comments and the comments were pretty good.

I even asked it how the script could be improved and it made suggestions around adding error handling and making some hard coded names into command line parameters.

I asked it to give me code to implement the suggestions and it gave me working code.

It’s much better than you give it credit for.

Sure, depending on what you ask and how that aligns with the content it was trained on and the word statistics it has learned, it can give correct answers.

OTOH I've also asked it what day of the week a given date was and received two different wrong answers depending on the exact phrasing of the question. I've also seen it confidently "explain" why taking 90% of a number and adding 10% of that back will get you to the original number...

The trouble is the output is a mix of truth and lies, and GPT has no way to distinguish between the two.

I’ll give you that,

I once asked it write a Python script that lists all of the accounts in an AWS organization with a given tag key and value.

It confidently, initiated the SDK (boto3) and the correct object on the SDK (Organizations) and then it called a none existent function - “get_accounts_by_tag”.

The next day I asked it the same question and it got it right using a technique that I would have never thought of.

On the other hand, I asked it “given the following XML file and a DynamoDB table with the following fields, write a Python script that replaces the value node in the file where a corresponding key is found in the table with the value in the value field”.

The code was perfect.

So in other words.

When OpenAI had a way for training live data all big marketing companies would produce a ton of information just to get their „facts“ to ChatGPT.

But as a user you can’t compare different sources like you would do on Google and you only have this BS answer which is fancy but tells you to drink dish cleaner because studies have found out that dish cleaner makes stuff clean and clean is healthy.

> What makes ChatGPT seem semi-intelligent

Its lack of intelligence is not the problem. High intelligence doesn’t preclude misinterpretation, mis-remembering, or overestimating it’s own understanding.

I think that depends on the goals of ChatGPT and/or what users are hoping to get out of it.

If it was just acting as a search engine using english as the query language, then lack of intelligence wouldn't be an issue - the quality of output would just depend on the quality of the source as we're used to with search engines.

However, what ChatGPT is actually doing - due to it's fundamental nature as a language model (dealing only in word/language statistics) is effectively combining information from multiple sources, which of course is potentially very powerful if it knew HOW to utilize these variously sourced facts to construct a correct answer... but of course it doesn't, so it'll happily generate content mixed from factual and fantasy sources etc, or correct textbook programming exercises with buggy code from beginners it dredged up someplace. It's not just mixed sources though - it's the intelligence of how to take a bunch of raw facts and deduce something from them, and of course ChatGPT is not a deduction engine.

I don't think it's a huge lift to restrict a language model to "known" good facts from search results. And to have it cite sources.

I made a proof of concept this weekend - https://github.com/VikParuchuri/researcher . There are some issues, but it's very useful.

Fascinating. This isn't at all how ChatGPT works. You're not leveraging the scale of the internet + style transfer to provide answers. Instead it's doing text summarization on search results.

It's a very clever proof of concept. Not exactly a large language model.

It's doing abstractive summarization over the search results, using GPT-3. The pipeline is:

  - Search using Google
  - Run some filters to exclude SEO spam, etc.
  - Scrape the pages that are returned
  - Find chunks of text likely to align with the answer (comparing embeddings)
  - Feed the most likely chunks into GPT-3 to get a summary

It is leveraging GPT-3 to produce better summaries, and it isn't purely extractive - the LLM uses context and knowledge to generate a better summary.

I want to experiment with a local model next, versus using GPT-3.

"There is no doubt that chatGPT is the future."

Why do you say this? It's going to be one of many similar products, even then, it's impressive, it's fun, but is it really useful yet? I think we have to wait and see?

By the way, I like ChatGPT and have a lot of fun with it.

You know that Wikipedia was highly unreliable in its early days, right? What ChatGPT needs if it's to be genuinely useful: https://xkcd.com/285/

It’s still unreliable. There’s no guarantee a given page that’s there today and good enough to serve a purpose won’t be considered not notable enough to remain tomorrow.

> The ability to cite sources

It doesn't need to cite sources, because it has learned to make them up on the fly.

Liars care about the truth; they want to subvert it. Large language models like ChatGPT don't "lie", they produce bullshit that has zero connection to what is or what isn't, but just sounds like something someone would say in that context.

It's possible, even likely, that ChatGPT and the like will help people formulate queries; but if it answers them and people somehow trust its answers, we're all doomed.

Reply on news.ycombinator.com