For example, I asked who the first president of Argentina was. It gave the wrong name (Juan Manuel de Rosas) with a half-true bio. I then asked it for books where I can learn more about this person.
It produced five books. None of the book-author pairings were correct, though three of the books were real and four of the authors actually wrote books. Of the four authors that existed, three actually did write about Argentina's history, while one is a professor of Spanish and gender studies.
This is my point: ChatGPT was trained to produce text, not to produce facts. What I'm seeing here isn't just something that can be optimized away, it's fundamental to the model's design. It was basically told to produce text that looks like it could have been written on the internet. Its text is really convincing: when I saw the list of books I thought for a minute it had actually done it! But it's still just inventing text that reads well, because that's what its training was optimizing for.
It could be paired with a different system that searches for facts, but then it's just a friendly abstraction layer on top of Google or some other system, not a revolution in search.
Other than that it betrayed no understanding of the material. It rather resembled a high school kid who had read a few too many books and spouted platitudes about any and all subjects without being aware of what was really going on.
That's only because it was trained on non-facts also.
But that holds for Google Search just as well ...
Google Search gives you links to content that might have the answer (excluding their own box). You can see the differences between the authors of said content.
ChatGPT gives you content that looks like the answer. You can't see the authors of the generated content.
And in either case, you're not solving the problem that commenters here are trying to solve: Google's bad results will still affect your search, so ChatGPT will be fed largely by dubious blogspam.
No, there’s a fundamental issue in that statistical language models don’t start with facts to form sentences, they form sentences based on previous tokens and statistical weights. The weights are not auditable or easily updatable.
[0] https://support.google.com/knowledgepanel/answer/9787176?hl=...
The lucky thing for Google if they got it working, is they are half there already. As much as I despise Google, I have to admit GPT won’t be their death.
I don’t think it is.
> Why can’t it have a mode where it returns more rigorous results, and includes some system of links as citation?
Because it is a statistical text-generation model; connection of results to specific sources is fundamentally not part of the way it works, so it can’t just be asked to supply them. You’d need a fundamentally different approach to do that.
There's also the ethical question. For example, duckduckgo will show you stack overflow answers. Google could easily do this but they have already gotten flack for displaying answers and taking away clicks from sites, which means reduced ad revenue for the resource actually being used. We could make even stronger arguments about services like ChatGPT in their current form, let alone the one we're talking about.
If we think a bit further in the future there's another extension that might not be what we want. Suppose our model contains all current human knowledge and can accurately give you answers and even produce paragraphs at will. This kills off a lot of websites. Is this a good thing? Suppose we incorporate an image generator along similar lines and now we have text and image search. That's a huge centralization force. There are a lot of great benefits to this that I don't want to downplay, but we should also think about if this is actually what we want because there are downsides too.
I thought I should also add that I had an at length discussion of these LLMs being Stochastic Parrots (an how we know they are) in a thread from 2 days ago https://news.ycombinator.com/item?id=34113011
So I think if you want to see more people with my dream then push towards a post scarce world. We'll still have plenty of resource constraints, but it does change the game a lot. I think you'll see more artists as well. A new renaissance. But I fear the transitionary period might be quite difficult and disruptive. I do think such a future has a fairly good probability of happening within many of our lifetimes and almost guaranteed if we actively pushed for it (or solved adjacent problems like HLI).
So don't be so pessimistic. There's millions of people out there that are interested in pushing the bounds of human knowledge because that's what they want. I'd argue that physicists and mathematicians wouldn't exist if this weren't true. They get paid shit for extremely difficult work. We could say the same about many sciences. While not all of these people are in it for bettering humanity (just for pursuing knowledge) it often aligns and there is a decent portion who believe this is the point of science and our purpose, as humans, in life.
In any case it can be trained to assert confidence via a side channel
I have read many similar comments to the grandparent, all saying the same thing, “I tried it, and it turned up wrong facts” all the while they forgot that they were deeply engaged with the content it was producing. Just think of how much our attention is monetized already, money will be poring into this space from here on. It’s only a matter of time.
Producing hallucinations or not isn't just a setting you can tweak in the model. The entire function of GPT-3 is to guess the next word from the words that came before. It's a hallucination machine that has been trained on so much real-world data and has such a large parameter set that for commonly known and discussed information it does a remarkably good job at creating factual sentences. But as soon as you get out into territory that it doesn't have memorized, it will do the next best thing and produce credible-sounding new material that may or may not be complete nonsense. Again, this isn't a setting you can tune, it's just what a transformer does. It models human language, not all of human cognition.
Also, since we can watch the model evaluation and see all its weights, there’s research that shows you can determine if any answer is a “retrieval” of existing knowledge or a hallucination.