Hacker News Reader

Comment by lolinder | Hacker News Reader

The problem with ChatGPT as a replacement for Google is that it was not designed to produce accurate facts, and it shows. This model cut its teeth writing articles about the discovery of unicorns in the Andes[0] for goodness sake! It's a language model, and a very impressive one at that, but language is used to express falsehoods and fiction just as regularly as it is used to express truth.

This doesn't mean that it can't produce accurate facts, most of the time it does! But when it does produce nonsense, it does it in exactly the same tone of authority, so if you don't already know the answer you may well walk away believing an AI hallucination.

And the trouble is it doesn't really matter if everyone here thinks "well, I would follow up each request with research to verify the answer", because most people won't! This is like the Google answer extracts, which fairly frequently mislead by extracting out-of-context quotes, except that there's no way to get the original context and there may in fact be no original context! This makes follow-up research much more complicated than with Google and therefore unlikely to happen. If ChatGPT replaces Google, the amount of nonsense on the internet will get even worse, which is something that until 2022 I never thought was possible.

[0] https://github.com/minimaxir/gpt-3-experiments/blob/master/e...

View on HN

Absolutely this.

I just did a discussion with ChatGPT on intuitionistic logic and it gave answers that were subtly wrong in my understanding of the subject. I have an MA in math but I couldn't be sure whether my impression of its wrongness was correct until asked it for a reference and it gave a nonexistent book by Per Martin-Lof, "Intuitionistic Logic", along with a list of actual books by him. It's an impressive and scary bullshit generator at this point.

And if someone can find a reference to Intuitionistic Logic by Per Martin-Lof, Oxford University Press, 1982, well, I'm wrong. But I went through Wikipedia, Google, Worldcat and Google Books.

No, you are probably right. It has no guardrails for generating a true citation. When you ask it for citations, its programming generates the form of citation. As the authors of the article discuss, it has no model of reality to understand that citations are not just a form of words and symbols put together.

Indeed and it gave several other actual citation with seemingly correct summaries.

And it's definitely less prone to obvious bs than just a few weeks ago. For a lot of general things, it gives a summary paragraph that's seems a distillation of standard information - the main problems seem to appear when these "scripts" lack consistency.

Edit: Once it "learns" to get citations right, you'll have to dig into those to see if it's right. It's easy to see that arms not looking good.

> Once it "learns" to get citations right, you'll have to dig into those to see if it's right. It's easy to see that arms not looking good.

But that's fundamentally not possible for the model as is, not in the sense you would normally think of citations at least. If it were to give an honest answer on "why did you say that Obama was born in Chicago", the model would generate training set example with propositions like Obama was born in ____, ___ was born in Chicago, ____ was ____ in ____ and so on - because that is how it learns.

Instead, to generate what you would actually want as citations, it would have to have a way to look up the statement it just produced in a corpus of "citeable materials" and find the closest match, based on some entirely different ML model most likely (presumably something similar to Google's search algos).

> As the authors of the article discuss, it has no model of reality to understand that citations are not just a form of words and symbols put together.

Which is the exact same issue that plagues it for the answer it’s generating a citation for.

I believe that it can be trained to improve this, and I believe it can be nearly solvable but I don’t believe it’s a solvable problem.

It can be trained to get facts mostly right using old-school academic tests and adversarial/supervised training, like an elementary school child. Today, it (supposedly?) punts math and some other logix to a separate function, and it just compiled the prompt. That can work for validating citations and “known facts” in many cases.

But again, it can’t “think” and understand when it returns something illogical (regardless of fact correctness).

The problem is mostly solved though. ChatGPT doesn't have the feature to cite its sources, but some other models do https://www.deepmind.com/blog/building-safer-dialogue-agents

I honestly doubt very much that a reinforcement learning based approach can scale to the breadth of facts that such a model would need to know about. Is there any reason that using a rewards-based approach to cover one set of facts would in any way generalize to a completely different area of expertise (say, rewards for true facts in mathematics generalizing to questions about literary criticism, or new world cultures), or would you have to include (manually curated) rewards for all areas?

It’s really good at showing what types of scientific books and papers should exist. GPT5 will just write them, so it isn’t lying.

It has no concept of lying.

GPT5 will.

ChatGPT in its current form can't replace google but plausible to think in its next few incarnations it will get there. Why can't it have a mode where it returns more rigorous results, and includes some system of links as citation? It can include footnotes for various passages of it's answer to indicate ambiguity, varying levels of confidence, etc.

This is the mental leap folks need to be making, the question is when will this type of configurably factual large language model be research benchmarked and productionized.

I have read many similar comments to the grandparent, all saying the same thing, “I tried it, and it turned up wrong facts” all the while they forgot that they were deeply engaged with the content it was producing. Just think of how much our attention is monetized already, money will be poring into this space from here on. It’s only a matter of time.

My argument isn't that I tried it and it didn't work, my argument is that an LLM alone fundamentally cannot work for reliable information retrieval.

Producing hallucinations or not isn't just a setting you can tweak in the model. The entire function of GPT-3 is to guess the next word from the words that came before. It's a hallucination machine that has been trained on so much real-world data and has such a large parameter set that for commonly known and discussed information it does a remarkably good job at creating factual sentences. But as soon as you get out into territory that it doesn't have memorized, it will do the next best thing and produce credible-sounding new material that may or may not be complete nonsense. Again, this isn't a setting you can tune, it's just what a transformer does. It models human language, not all of human cognition.

ChatGPT isn’t trained on next word prediction alone, it was trained on human feedback. The evaluation is next word search, but the model has evolved to give “correct” answers when evaluated that way. So it’s not just the most likely answer anymore.

Also, since we can watch the model evaluation and see all its weights, there’s research that shows you can determine if any answer is a “retrieval” of existing knowledge or a hallucination.

For the same reason why it might produce a bogus answer: it's a language model, not a general AI. Prompt it for footnotes and it will generate footnotes for books that don't exist or that don't actually contain the citation. They'll match the form of a footnote, because that's what the model was trained to do, but the citations won't resolve.

For example, I asked who the first president of Argentina was. It gave the wrong name (Juan Manuel de Rosas) with a half-true bio. I then asked it for books where I can learn more about this person.

It produced five books. None of the book-author pairings were correct, though three of the books were real and four of the authors actually wrote books. Of the four authors that existed, three actually did write about Argentina's history, while one is a professor of Spanish and gender studies.

This is my point: ChatGPT was trained to produce text, not to produce facts. What I'm seeing here isn't just something that can be optimized away, it's fundamental to the model's design. It was basically told to produce text that looks like it could have been written on the internet. Its text is really convincing: when I saw the list of books I thought for a minute it had actually done it! But it's still just inventing text that reads well, because that's what its training was optimizing for.

It could be paired with a different system that searches for facts, but then it's just a friendly abstraction layer on top of Google or some other system, not a revolution in search.

That makes sense. The only thing about gpt that impressed me was when I asked it to condense an answer into one sentence, it was able to do it.

Other than that it betrayed no understanding of the material. It rather resembled a high school kid who had read a few too many books and spouted platitudes about any and all subjects without being aware of what was really going on.

OTOH I could imagine a hybrid situation where the AI Bot searches Google for you and restates the millions of results Google gives it

This was my immediate thought as well. I don’t need millions of searches done, I need 20, then a language model to look through them and collect text that helps me find what I’m looking for.

The lucky thing for Google if they got it working, is they are half there already. As much as I despise Google, I have to admit GPT won’t be their death.

I agree that a hybrid could work, although ChatGPT struggles to keep context with more than a dozen paragraphs, so I doubt it could summarize millions of results.

And in either case, you're not solving the problem that commenters here are trying to solve: Google's bad results will still affect your search, so ChatGPT will be fed largely by dubious blogspam.

Even in a small number of paragraphs language models hallucinate. For example Barack Obama, it’s not hard for me to trick it into saying he was born in Chicago. He was not. But there’s a ton of text about Barack Obama in Chicago. Plus, Michelle was born there. When predicting the most likely next token for “Barack Obama was born in ______” there’s a lot of confounding text.

No, there’s a fundamental issue in that statistical language models don’t start with facts to form sentences, they form sentences based on previous tokens and statistical weights. The weights are not auditable or easily updatable.

And any kind of "tuning" of the weights to get better results is going to be so bespoke that you might as well just curate the response yourself

So LLM's are pretty much Mark V Shaney at heart, only with enough weights to get to the far side of Eliza?

Though for that matter a ChatGPT based product would be even more vulnerable to the same thing that made Google search suck: content sources gaming its process

Google already does this, it is called Knowledge Graph[0].

[0] https://support.google.com/knowledgepanel/answer/9787176?hl=...

> This is my point: ChatGPT was trained to produce text, not to produce facts.

That's only because it was trained on non-facts also.

But that holds for Google Search just as well ...

I think there's a difference in presentation.

Google Search gives you links to content that might have the answer (excluding their own box). You can see the differences between the authors of said content.

ChatGPT gives you content that looks like the answer. You can't see the authors of the generated content.

Do you think that ChatGPT can't be trained to output links to sources?

Because to make it focus on always producing accurate results (or rather a very low false positive rate) you need to dramatically change and tune the model. Which is much harder than what Google does because it means you'd also need to vet a lot of facts as well (by weighting certain sources, not individually). Or allowing it to search the internet (which also comes with other issues and if it is using Google to do that then it doesn't really kill Google). I do think Google's knowledge graph is more in line with what you're actually thinking. Lots of Google searches can be done without actually clicking links.

There's also the ethical question. For example, duckduckgo will show you stack overflow answers. Google could easily do this but they have already gotten flack for displaying answers and taking away clicks from sites, which means reduced ad revenue for the resource actually being used. We could make even stronger arguments about services like ChatGPT in their current form, let alone the one we're talking about.

If we think a bit further in the future there's another extension that might not be what we want. Suppose our model contains all current human knowledge and can accurately give you answers and even produce paragraphs at will. This kills off a lot of websites. Is this a good thing? Suppose we incorporate an image generator along similar lines and now we have text and image search. That's a huge centralization force. There are a lot of great benefits to this that I don't want to downplay, but we should also think about if this is actually what we want because there are downsides too.

I thought I should also add that I had an at length discussion of these LLMs being Stochastic Parrots (an how we know they are) in a thread from 2 days ago https://news.ycombinator.com/item?id=34113011

What we want is not what we get unless it makes bank. Nobody (generalizing here) works to better humanity, they work to get money. if it aligns well that’s just shear luck. The world needs a lot more people who want see the world to succeed.

I think this is a highly pessimistic view. While I know many that are profit motivated and I won't argue that this isn't the majority, I know many people that just want to pursue knowledge for the sake of knowledge. I'll tell you that personally my dream is to just be able to research and not worry about constraints like publishing (I'd rather open blog) or computational constraints (don't need a data center. A few dozen A100s is more than enough). There are a lot of people like me and a lot that would be like that if they didn't have financial constraints. Having this dream is a privilege and even I didn't have it when I was living paycheck to paycheck.

So I think if you want to see more people with my dream then push towards a post scarce world. We'll still have plenty of resource constraints, but it does change the game a lot. I think you'll see more artists as well. A new renaissance. But I fear the transitionary period might be quite difficult and disruptive. I do think such a future has a fairly good probability of happening within many of our lifetimes and almost guaranteed if we actively pushed for it (or solved adjacent problems like HLI).

So don't be so pessimistic. There's millions of people out there that are interested in pushing the bounds of human knowledge because that's what they want. I'd argue that physicists and mathematicians wouldn't exist if this weren't true. They get paid shit for extremely difficult work. We could say the same about many sciences. While not all of these people are in it for bettering humanity (just for pursuing knowledge) it often aligns and there is a decent portion who believe this is the point of science and our purpose, as humans, in life.

While I see what you are saying and there are people doing the good work, when I read of companies circumventing the Russian embargo knowing full well their stuff gets turned into weapons, and people without thought buying the cheapest shit from amazon and alibaba made in dire circumstances, I am not so sure, or at least, do not see enough awareness for a treshold changing outcome. Some play the economic game on a criminal level, and some profit without giving back and without scruples.

> ChatGPT in its current form can’t replace google but plausible to think in its next few incarnations it will get there.

I don’t think it is.

> Why can’t it have a mode where it returns more rigorous results, and includes some system of links as citation?

Because it is a statistical text-generation model; connection of results to specific sources is fundamentally not part of the way it works, so it can’t just be asked to supply them. You’d need a fundamentally different approach to do that.

Contrary to all the naysayers I think a future version of GTP will be trained to evaluate its own certainty and express this verbally. Probably through an extra training channel.

It has no concept of certainty, unless it was embedded in the original data. What you are describing is an more general AI. This is a language model. I like to think of it of a cognitive intelligence: the ability to process input and compare it to a stunning array of trained data that will have many parallels to reality but has no concept of it.

It can be trained some form of trustfulness, and a measure of certainty is intrinsically part of the model (compare with image to text models etc "70% cheetah 10% dog")

In any case it can be trained to assert confidence via a side channel

devils advocate, but Google wasn’t designed to make sure the links it aggregates at the top are accurate/isn’t just blog spam designed to cater to specialized SEO tricks

I agree that Google is terrible at its job and produces garbage most of the time. That's the only reason why an LLM chatbot is seen by anyone as a viable replacement.

But at the very least Google always gives you the source, which allows you to read it for yourself and judge its trustworthiness. No such option for ChatGPT.

> Google is terrible at its job and produces garbage most of the time

Are we using the same search engine? I use DuckDuckGo most of the time but Google is hardly terrible, even if it could be improved.

I find what I’m looking for in the top results 90-95% of the time. The other 5-10% also contains times where there simply isn’t a good link that solves what I’m looking for even after digging hard.

> Are we using the same search engine? I use DuckDuckGo most of the time

Then you are not using the same engine!

(I also use DDG and find its results superior to google).

DDG is much more straight forward vs Google often tries too hard and fails.

It's like how Twitter's default feed changed to try to tell you what you wanted to see, while the old chronological feed was the content you originally asked to follow. The simpler feed was always better and less contentious for non-newbie/casual users.

Google is better at deciphering what you want, but DDG is straight up: here's the simplest results based on your query.

Maybe the solution is both? An evolution of the ML approach that Google uses (ala ChatGPT style mimicking a human) AND offering a tab, or box, or whatever interface, with 'simple' straightforward old-school search results. Until the former is better 99% of the time, they shouldn't abandon the latter.

The highest placed links on Google are basically guaranteed to:

1) Not be the actual source of content 2) Not link to the original source

For most programming questions at least, the highest placed result links are StackOverflow answers related to your query (if any exist, of course). I would say this alone invalidates your "virtually guaranteed" claim.

I was pleased at first when I tried to use ChatGPT as a search engine. I asked something like "give me the top 5 blog posts to learn about C++ metaprogramming" and it actually spit out extremely plausible titles from top people (e.g. Dave Abrahams, Eric Niebler, Herb Sutter, etc), but when I said "no please give me the actual URLs", they were all 404.

If Microsoft somehow is able to implement the information synthesis (e.g. "give me an example of C++ metaprogramming, using Boost Hana") with actual indexed links from sources, that would probably be the one thing I'd see in the near-ish term that could upset the search engine balance of power.

ChatGPT gives me the urls when i asked a similar question for go programming. My guess is its an engineering integration in the nascent stages.

I don't know if this is feasible or not, but having the chat bot also be able to respond with a subset of sources that it based its answer on would be a satisfying combination of both worlds I think.

I would be really curious to see how trying to learn a new subject via a chatbot that is able to also provide source material links would go.

> I don’t know if this is feasible or not, but having the chat bot also be able to respond with a subset of sources that it based its answer on would be a satisfying combination of both worlds I think.

But an LLM doesn’t look at particular sources, find relevant ones, and write a response based on them, it is a mathematical summation of the way all of its sources look as sequences of text, and produces a response that based on that fits with its prompt. Identifying which sources most influence a response isn’t merely something that they don’t currently do, its information that, AFAIK, there is no plausible way to recover from the model.

After I commented I did do a little bit of thinking about the little that I know about this subject and realized that it probably isn't feasible to extract that data. Thanks for the comment because... you are right as far as I know. It is a bit disappointing to realize this though.

I can't count the number of times I've clicked through to a link expecting to find more context than the excerpt on the search page and that content isn't there. Instead I get trapped in a sea of circular SEO spam that all promises to give me the answer I originally searched for but ends up leading me nowhere except to other article summaries with content I didn't even want to read about. Where's the source?

A good chunk of the remainder of the times I click through - especially on work-related stuff - it turns out the page is some kind of cribbed content from GitHub, StackOverflow and so forth, but mangled together and reskinned with different randomly-generated usernames making the "comments", so it seems like it's a different site, and you can only really tell by picking a specific sentence that appears unique and then manually searching for that same sentence on several different sites. No clear source there either.

Web search is broken, and it has been for several years now.

> I can't count the number of times I've clicked through to a link expecting to find more context than the excerpt on the search page and that content isn't there.

Excellent: this quite quickly tells you that the search result was wrong. So you quickly got very relevant context and can tell at a glance that there is no source. You don't get any such context when GPT provides an invalid answer.

In the future, the source might be written by ChatGPT anyway.

Actually it was designed for that, and it was top line OKR for a long time. Amit Signhal rewrote the search engine and was a great leader caring a lot about relevancy until he was kicked out for harassment (it was deserved as far as I know, but at the same time Google lost an amazing engineer).

In my experience the detoriation came after the change in leadership.

It is important to recontextuilize harassers. They are typically parasites on the organization, destroying others by stealing their work and eliminating others from the work force.

Undoubtedly he came off as clever, but being a harasser means that his achievements were outweighed by the benefits.

It's dangerous to say

"he's a harasser => he must be a parasite on the organization"

because then you end up with

"he's provably an excellent worker and hugely productive => he can't be a harasser, you're mistaken/lying"

Some people are really smart and productive and a credit to their profession and also awful to women and/or minorities, and you'd never know if you didn't witness/experience it.

"Some people are really smart and productive and a credit to their profession and also awful to women and/or minorities"

But if they are awful only to white straight males, they are ok?

Recontextualizing harassment as stealing someone's work is not just inaccurate, it's degrading to harassment victims and not accepting that somebody can be a bad person in some situations (on offsite when he's drunk) and be an amazing engineer at the same time when he's sober at the workplace.

Harassment is harassment, stealing someone's work is stealing someone's work, let's keep changing the meaning of words to politicians.

I think it's a big mistake to assume that if someone is toxic, they must also be stupid/incompetent/generally useless, all evidence to the contrary. If nothing else, it will make you consistently underestimate some pretty dangerous people. It's important to know your enemy, especially their strengths.

they aren't necessarily more or less competent, just don't mind also stealing credits from others to boost their ego, value, bank account or whatever

I believe this to be a huge deal. In no time a huge part of the internet will be AI-generated and we will pollute our data set for research and training AI with things that are subtly or completely false at times and we won't be able to tell.

It's a lot like iRobot or other stories in which the robots will start to subtly misbehave, like a program you can't debug.

How is this different than regular intelligence? We grow up trained on a polluted dataset, being taught things that are subtly or completely false, and we often struggle to tell which things we have learned are false.

And what was the result of unchecked faulty learning on a corrupt dataset by natural intelligence?

Dark ages where millions died in pointless wars over who's imaginary diety is the true one. A third of europe dying of preventable plague. It took a rigorous system of citation and a publically funded system of education to crawl out of the drakness.

Should we try AI powred dark ages but with resource shortage and nuclear weapons?

Once in a while somebody does some actual on-the-ground research and people are convinced to change their minds through evidence and reasoning. So far AI can't do that, it'll just parrot whatever it hears more often, (apparently) with some degree of randomness.

Maybe it'll learn to adopt a new view, but for now it won't be through actual reasoning.

GPT has had up/downvotes for years. They were used to train ChatGPT. It’s not just “what’s the most likely answer”.

(The most likely answer is the letter “e”.)

It is true but then again, we constantly ajust the trust in sources based on what works and what doesn't.

If I watch a youtube tutorial teaching me how to bake pizza and it comes out totally wrong and a watch another that produces exelent pizza, I disregard the first channel.

And there are many more ways where humans check the reputation of sources automatically.

GPT currently lacks this possibility.

Well this is unfair because we haven’t given natural intelligence a chance yet.

What if we invented an algebraic notation for search indexing and allowed people to index and classify documents in their own way. Then a search engine could do something special with all that.

Human intelligence cannot progress without a notation to capture our thoughts in.

Are you aware of https://commoncrawl.org/ ?

> This is like the Google answer extracts, which fairly frequently mislead by extracting out-of-context quotes

My favourite Google extracts:

Q: Who invented Running

A: [random guy] invented running in 1865 when he tried to walk twice at the same time. It was an overnight success

Q: How much does a [model of a bycicle] weight?

A: 140 KG (Maximum weight of the rider)

Q: Define an english Person

A: A Cunt

1/3 isn't bad.

Truthfully the internet is already full of crap a lot of it by design by different interest groups, countries spreading fud about other countries political parties doing the same for their opposition the web has become a cesspool. Majority of the people I personally assume at least 9 out 10 don't have capability or care to do the research to sift fact from fiction in the first place so this probably won't change anything. I have been trying to make it work for me and teach me some python scripting or creating regexes. If you question it correctly it outputs out something even if you might not be able to run the first time generally it works. I fed it 2 csv files/text asked it if it was possible to go convert 1 format to another got a python script that I was able to convert large number of files. Now scripting and programing might seem trivial to large parts of hackernews audience but it will be a huge change for people in general. From what I feel the AI keeps improving its answers is it being trained on the million of responses it get on the answers it gives.

Google was not designed to provide accurate results either, and it shows. I can find results to confirm every conspiracy theory I can think of. ChatGPT at least denies conspiracy theories.

In search results, you can at least evaluate how trustful the document is, based on various factors. This resembles how academia has evolved: you can dig into all the references, see the author's publication history and even their reputation. How would you do this with ChatGPT? We can't even verify the sources of its claim and it sometimes even makes up non-existent references in a very confident tone.

These are early days. Everything is going to move faster than you can keep up. You'll see head-turning headlines on a daily basis.

Next year will see AI begin to disrupt so many businesses you'd thought were absolutely impenetrable. I don't think anything is safe anymore.

By the end of the decade, I think even manufacturing and robotics will be automated. Everyone from McDonalds to Apple and Ford will feel the pressure.

I've got a mind-blowing demo coming out soon that will put even more fields on notice. And that's just my team. There are so many people working on AI-first products now. Incumbents will be overwhelmed and washed away.

(If ever there was a cue for a career change or opportunity to make a whole lot of money, it's this. Don't ignore this one! You're still early enough to take advantage.)

I'm not questioning the AI revolution, I'm questioning the applicability of large generative language models to information retrieval. We need a different kind of model for that, not just a few tweaks here and there to this one.

Given everything we've seen in the last year alone, I'd not gamble against that kind of model being just around the corner.

I can think up architectures I could draw on a napkin that could do this. Focused research teams will nail this.

It's coming.

/r/iamverysmart

>If ever there was a cue for a career change or opportunity to make a whole lot of money, it's this. Don't ignore this one! You're still early enough to take advantage.

Any suggestions on how to take advantage?

Dive into ML development?

Learn how to write prompts for AI engines?

Invest in AI companies? Sadly OpenAI is private, but Microsoft is a monetizing partner I guess (Copilot)

I'd take a lot of this with a fairly large helping of salt. This poster obviously has a vested stake in this direction of technology working out.

> (If ever there was a cue for a career change or opportunity to make a whole lot of money, it's this. Don't ignore this one! You're still early enough to take advantage.)

Details?

Well, there's one thing that humans are much better than robots for the forseeable future: Energy efficiency. Try building a robot that contains a general purpose AI and the motoric functions and sense as humans that runs on the energy of three meals.

Now of course humans want and need more than just 3 meals but still pretty good.

But if most of your work consist of interfacing with computers, you might be in trouble.

Also, since humans aren’t slaves, they do their own maintenance. And they don’t do exactly what you ask, which is good (when they do it’s called “working to rule” and is a kind of strike).

It’s very rare for automation to cause unemployment to the point it possibly never happened.

> Next year will see AI begin to disrupt so many businesses you'd thought were absolutely impenetrable

Why, because of LLMs in particular? What is the ground-breaking research that will be fueling an AI revolution?

> There are so many people working on AI-first products now. Incumbents will be overwhelmed and washed away.

Incumbents like Google have been doing AI for ages. Microsoft has a very important investment in OpenAI

> By the end of the decade, I think even manufacturing and robotics will be automated. Everyone from McDonalds to Apple and Ford will feel the pressure.

McDonald's, Apple or Ford sell products, not manufacturing services. They will be delighted if manufacturing is automated.

It seems that at the current velocity of improvement it is only a matter of time - very near future - that those problems will be mostly worked out. ChatGPT already outperforms Google search in many ways. For example I was doing some research on Scandinavian history and found the results far superior on ChatGPT, while I had to sift through pages of junk on Google. I took the results with a grain of salt just like I would with Wikipedia and verified each point, and found it to be very accurate. With this I can cut my research time down from hours to minutes on a problem.

Google links are notoriously inaccurate. Google was always great at finding relevant/clickable links- they never tried to direct you to factual links.

I can't wait for a few years, when ChatGPT is replaced by something 10^x more powerful, and then comments like this look like nonsense of someone who lived in a past era just like we now similarly make fun of comments from people that made in past eras about how certain technologies, criticized in their current state without the ability of the person to look ahead. Even though the person you're replying to mentioned ChatGPT, ChatGPT is just the FIRST of many. Imagine somebody saying in 1803 "Car is unusable because you have to keep filling the coal, it's just not convenient, and a horse will always have a future."

I don't think anyone who has played enough with ChatGPT is championing the notion that ChatGPT itself will replace Google. They are more so steel-manning the argument that Google might not be the best interface for search.

> The problem with ChatGPT as a replacement for Google is that it was not designed to produce accurate facts

But chat bots could be designed to do that, no?

Are there any siginificant technical barrier to prevent so?

Yes: accruing a large and diverse enough training set where true and false facts are correctly labeled. The GPT models require enormous amounts of text to be trained on, so manually adding true/false labels to every part of every sentence is impossible (and automatically doing it would require exactly the AI we are trying to train).

A revolution in this area would require some entirely new training algorithm that can reach GPT-levels of fidelity while using magnitudes less training material. An even bigger GPT model trained on even more text is not going to do much in approaching a solution to this problem.

Google was not designed to give you accurate facts either, it was designed to calculate popularity of links based on backlinks and then show you the most referenced links first.

> This model cut its teeth writing articles about the discovery of unicorns in the Andes

This is a different model, called GPT-3, released in 2020. ChatGPT has the same number of neurons, but is trained with both human feedback and web data, rather than just web data like GPT-3.

What you're saying still holds up, though -- ChatGPT still hallucinates information.

Isn't ChatGPT just a conversational interface for GPT-3?

Yes, the human feedback model is called InstructGPT and has been the official “GPT-3” for some time now. ChatGPT is only somewhat more trained, but unlike the playground, it’s free and has a Markdown renderer.

No, it has different training, so entirely different neuron connection weights. They are very similar though.

Search engines are not designed to produce accurate facts either.

Verification of facts requires direct experience through careful experimentation.

ChatGPT’s responses can either be verified or they are as unfalsifiable as any other confidently expounded conjecture found on your computer screen.

get the journals open source and add it as a module, although you'd have to strip out the unreproducible social science studies but that could be a good thing.

Neither was Wikipedia. But it works well enough.

Wikipedia is designed to be as accurate as possible and is one of the few remaining website that produce value and content.

Please notice that in current setup of 'Internet capitalism' people that produce value and content earn nothing, and people that harvest that content for profit or engage in outright fraud make fortunes.

I wasn't aware Google produces accurate facts. Cf. front page links to Wikipedia as an obvious counter.

1. Provide citations like you.com already does. 2. Add Trustrank/Trustflow into the training set

Chatgpt is is beta, we can safely assume the situation may change from now to 2024/2025.

Not at a fundamental level. Even if you exclusively trained one on some reasonably authoritative source like Wikipedia it would still happily tell you complete nonsense.

These models have no way to differentiate that elves are fictional while Indians actually exist.

What makes you think Google is good at finding facts?

RIP google. There's a joke that the best place to hide a murder is on page 2 of google search. The fact is GPT has at bare minimum parsed the top page of google and reasoned about the top results to give the best answer with as much context of why it reasoned so. GPT is basically a superset of google with added reasoning. I can't see it giving worse results than Google. Both will produce a bogus answer if you ask a bogus question, but GPT will never produce a bogus answer when google has the correct one.

I think it's really problematic to use the word "reason" to describe what GPT does. As others have commented in this thread, GPT is a language model, which in no way reasons about anything it writes. It is simply writing whatever fits statistically to the prompt and context. Attributing reasoning or too much intelligence to this could prove quite dangerous.

> As others have commented in this thread, GPT is a language model, which in no way reasons about anything it writes. It is simply writing whatever fits statistically to the prompt and context.

Those people are wrong; statistics are a kind of reasoning (intuition) and if you try it, it obviously does the other kinds of reasoning too. I have all kinds of answers from it that are logically coherent and can’t possibly have appeared on the internet already.

(Try asking it to write new SCP wiki entries.)

Just because some of its answers are logically coherent due to it following statistical patterns, does not mean that the model is reasoning. Reasoning implies some type of discipline which prevents nonsensible arguments (even in uncommon situations).

No such thing as “is reasoning”, only “does reasoning”. Are you an unreasoning being if you make a math mistake?

You can see it “does” reasoning if you say “think step by step” after a question. It can attend over its steps then.

> but GPT will never produce a bogus answer when google has the correct one.

What are you basing this assumption on? GPT can always generate a bogus answer, because its only notion of whether an answer is good or not is whether your prompt + that answer are statistically similar to text in its training corpus. The more text from the training corpus it resembles the better GPT will think the answer is. For more obscure topics, this means that the chances of it hallucinating based on similar sounding much more popular topics increases greatly.

This way of working also means that facts that appear more in the training corpus have a higher chance of being part of the model's outputs. For example, if the training corpus was not carefully curated to exclude vaccine conspiracy theories, there would be a good chance that the answer GPT gives to "autism is caused by ___" would be "vaccines", as this would likely be the most common answer in its training corpus. In contrast, Google's algorithms have more sophisticated biasing especially for these kinds of sensitive topics to ensure that more high-confidence sources are likelier to be top results, even if they are a minority of all the pages on the internet on that topic.

> GPT can always generate a bogus answer, because its only notion of whether an answer is good or not is whether your prompt + that answer are statistically similar to text in its training corpus.

“Statistically similar” is doing a ton of work even for the original GPT model - just printing the letter “e” is “statistically similar” to English text, if it can even be grammatically correct you should be surprised.

And it’s certainly not true for the last two generations (InstructGPT and ChatGPT) which weren’t trained with that objective at all but used human raters.

This comment is based on such a large amount of BS. Reasoning over facts, as a concept, is completely divorced from the function of these GPT models.

Reply on news.ycombinator.com