Hacker News Reader

Comment by pclmulqdq | Hacker News Reader

I'm just going to put this out there: chatbots for search seem to be barking up the wrong tree, at least for the next few years. People expect searches for factual information to be inaccurate probably 1 in 10,000 times. A chatbot literally cannot guarantee anything better than about 1 in 50. Keyword matching plus authority scoring was unreasonably effective at this. There are so many other applications, but execs at Google and Microsoft are obsessed with re-creating "the computer" from Star Trek.

View on HN

> People expect searches for factual information to be inaccurate probably 1 in 10,000 times.

Do they? I saw someone recommend using ChatGPT to learn how something worked earlier today on HN. And there were replies that agreed.

A ton of people seem excited about AI search. Google is bad enough without it at this point. But to me it feels like those helper chatbots that appeared everywhere a few years ago, redux.

Sure they have a more realistic tone and can understand questions far better than basic pattern matching. But it still doesn’t work.

I switched to DuckDuckGo about 2 months ago. The results seem about 95% as good as Google for most things, without all the ads and nonsense.

I have a hard time seeing today’s AI, impressive as it is in many ways, making search results more accurate. If they wanted to do that I feel like Google has a ton of low hanging fruit to fix already instead of hasty “features” to try and not look “behind”.

I just want to point out that ChatGPT is not an AI search product. It is a little like a "rubber duck" in my usage, and that is pretty helpful, but not something you lean on for factual research. Some other people use it as a copywriter or editor, which is another good use.

I agree. But I seem to keep seeing people trying to use it like a search engine or a Q&A site where you can just ask it whatever you want to know.

But it can’t do that.

It can do a lot of other things, useful or just fun. But it’s not a virtual professor that knows the correct answer to any question.

In some areas it does a good job of giving me a better answer than me checking 5-10 random stack overflow and blog posts with different answers over 20 years...

But, it is completely incapable of saying "I don't know the answer to that" and will spew the wrong answer to anything. For fun I have arguments with ChatGPT that it is just making things up and it should just tell me "I don't know" and it will happily say "I'm sorry, you are right. Here is the right answer <insert made up fact>" and it will do so generating random garbage indefinitely.

The problem is that I can't trust it to provide facts any better than asking my bull-shitting co-worker who is convinced he is right about everything and when called out says "yeah, that's what I was saying, <insert contradictory thing>"

How is Google bad for most situations? People say this so often here and make comparisons to the past. What is the issue?

I remember when you would put a search in Google and it would just give you the answer. Almost psychically, the number one link would probably be what you wanted. Maybe number two.

Today, first of all, you have to be careful to ignore all the fake search results that are actually ads.

Then there’s the problem of spam sites. References to what you want from Pinterest, but no way to get to the original source. Copies of Stack Overflow answers (or whatever primary source) on some scam domain that steals content to show ads. But it ranks very close to the primary source. Gotta ignore that.

My new favorite is Google ignoring what I’m actually searching for. I can search for “objective-c library to do X without Y”.

Google will helpfully give me pages and pages of “swift library to do X”. That is a much more popular search, so clearly that’s what I meant. Forget Objective-C. Forget without Y. Just dozens of tutorials on the far easier/more popular thing I’m NOT doing.

At this point in many ways it’s worse than the old keyword-based search engine like HotBot. They at least tried to find what I asked for instead of guessing what I probably meant based on popularity.

Don’t get me wrong, it’s kind of amazing that I can search Google for “the female comedian who was in a movie with a dinosaur” and the first result is about Tammy and the T-Rex, a truly awful movie with Whoopi Goldberg.

But that’s offset by seemingly simple searches that go sideways for no discernible reason.

I want 2007 Google back.

Edit: and don’t forget the ways they bent the market. Why does every recipe have a 5+ paragraph story to go with it? Because without that Google penalizes the page for having “no content”. So now all recipe sites are terrible, because they have to be to get visitors.

> I want 2007 Google back.

The SEO mess was indirectly caused by Google PageRank and other related optimizations.

Maybe we want 2007 Internet instead?

The AI world will inevitably lead to "content optimization", so the Chatbots that will be asked "questions about life" (as of Today, where people usually search on Google "I have fever/depression/a turbulent child/tomatoes in my fridge, what should I do?") will more frequently answer specific products.

Instead of promoting a single website (Current SEO strategy), content producer will be tempted to produce many texts on a great variety of sources, to reinforce the model on a specific subject.

Time will tell if in 2035, we will want 2023 ChatGPT.

You’re right.

I meant something along the lines of “I want Google to be as effective today as 2007 Google was in its era.”

I know Google and especially their ads are responsible for a huge chunk of what I dislike about today’s Google and the web.

ChatGPT 2026 being trained (probably unknowingly) on the output of ChatGPT 2023 is going to be ‘fun’.

> Don’t get me wrong, it’s kind of amazing that I can search Google for “the female comedian who was in a movie with a dinosaur” and the first result is about Tammy and the T-Rex, a truly awful movie with Whoopi Goldberg.

Google actually gets this one wrong. "Tammy/Tanny and the T-Rex" starred Denise Richards and Paul Walker. You're thinking of "Theodore Rex". Both were in fact truly awful, although I have a soft spot for the dinosaur making a phone call gag.

Try Kagi, the search quality is far superior to DDG. But you've got to pay for it :/

ChatGPT is fundamentally something like AI-assisted interactive Web search because it was trained on text from the Web.

In my opinion, "AI-assisted interactive Web search" absolutely a killer app for LLMs, and will be a huge step forward in our ability to quickly consume the huge amounts of information out there. The fact that it's a "chat" is useful only insofar as the chat history gets absorbed into the prompt, and it allows you to advance the search incrementally and iteratively using natural language queries.

The fact that the output is designed to feel "chatty" to some extent is immaterial. Chatbots, as you point out, are not a killer app, and fixating on the "chat" aspect as anything other than a means of interactivity is missing the point. People mostly don't want or care about simulating real human interactions in general. They just want more useful, more efficient interactions with their machine. Any perception of chattiness is best used in order to serve that goal, and should not in my opinion be a goal in and of itself.

That said, it's a shame that Ask.com isn't running this, otherwise we'd be able to call it Ask Jeeves 2.0. Now we have... Bing Chat? Poor Cortana!

What does ChatGPT have to do with "AI-assisted interactive Web search"? Did they roll out something new?

Bing is basically ChatGPT-4 with search retrieval integrated.

Bing has essentially stopped working for me, and I'm mad.

Mad because (pre ChatGPT) I had been getting better results for a little while than with Google, and also mad because I expect Google will soon screw up their search engine in the same way.

What I'm referring to is that popular queries suddenly seem to have a much larger "shadow" making it impossible to search for anything remotely similar.

This was already a degenerative disease degrading Google search, but what I assume is the AI infusion to Bing seems to have taken it to a new level.

> What does ChatGPT have to do with “AI-assisted interactive Web search”? Did they roll out something new?

The GPT-4-powered Chatbot agent in Bing. But I think GP is pointing out that, since the models underlying ChatGPT (both GPT-3.5 and GPT-4) are trained on a wide corpus including substantial content from the public web, even when they aren’t integrated into something like the new Bing, they are inherently a kind of (potentially somewhat stale) AI web search on their own.

I think people are way more skeptical than that - even non-technical users are onto content farms - but you are being generous by not mentioning the time horizon problem.

A search engine which doesn't not know anything that's happened in the past 18 months is not useful.

while that's true the bigger issue is that the conversational interface of chatbots is uniform so you as the user have no ability to at first glance get any indication of whether the source is trustworthy.

When you Google and you get two links, one from the YT comment section and another one from the Stanford Encyclopaedia of Philosophy most people who want factual information have a decent idea what to click on. Even when these chatbots provide sources it seems like half the time it's just made up.

Putting a game of telephone between you and the source with the guy in the middle being completely indifferent to reality by design just seems bad.

>People expect searches for factual information to be inaccurate probably 1 in 10,000 times.

It's okay they've recently cooled this off with google's 'answers' or whatever they are called that are placed on top of the results which are almost never right

I think going on an exciting journey with a really intelligent chatbot that is suffering from Dunning-Kruger effects is going spice up things on the web.

Imagine searching for symptoms! What an emotional rollercoaster awaits.

Dunning & Krueger’s paper showed that self-assessment correlates positively with performance, at least for the small number of tasks and undergrads they measured, so this joke doesn’t land the way you hoped. Apologies for the nit, but DK is almost universally misunderstood (and the authors are complicit in encouraging the widespread misunderstanding). That paper and its aftermath represents a lot of what’s wrong with social science and popular conceptions of it, so I take every opportunity I can to dissuade people from citing (or mis-citing) DK.

It's not really a citation. "Dunning-Krueger" has become shorthand for people not knowing what they don't know.

Kleenex and Xerox sympathize.

It certainly is a citation. What makes you say that referring to a specific paper via the name of the authors isn’t citing it? That’s the definition of the word. It’s not a formal academic citation, but it is a citation.

I’m well aware of the shorthand, it was the subtitle for the paper (“unskilled and unaware of it”), and that is more or less exactly what I’m talking about, because it’s not generally true. People also tend to claim that excessive confidence signals low competence in the name of DK, which may be implied in the comment above. Both of those concepts are basically incorrect, DK didn’t broadly measure what people know or don’t know, they used 4 narrow tests, one of which was subjective(!). Nor did they measure confidence, and when it comes to normal topics that people get college degrees in, followup research has shown the effect disappears and sometimes reverses [1]. The DK paper essentially measured how well people can guess their rank amongst a small set of peers (Cornell undergrads), which unsurprisingly isn’t perfect because it’s a guess. This is why DK is probably only poor methodology measuring a regression to the mean [1][2], and not uncovering anything interesting about human behavior.

[1] https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-...

[2] https://en.wikipedia.org/wiki/Dunning–Kruger_effect#Statisti...

It’s just better to not mention DK in the first place, it’s not something you can observe on a single speaker outside of a statistical study, and it simply doesn’t apply to all topics and all people, even when one does understand what the paper did show.

>It certainly is a citation

That, to me, is a statement about someone else's intent.

I have no idea what you mean, would you like to elaborate?

I believe a formal citation (APA style) looks vaguely like this:

  gonzo41. (2023, March 22). Bard uses a Hacker News comment as source to say that Bard has shut down (By cryptoz). 
        Hacker News. Retrieved March 26, 2023, from https://news.ycombinator.com/item?id=35256223

Sorry if I screwed it up; that is just for the overall look.

I'd agree that a citation doesn't have to be formal, but if it isn't, then the meaning of a passing mention of names has to be more inferred from context.

Right, we’re not talking about a formal academic citation, so the format is irrelevant in this context.

Why does it matter if the meaning has to be inferred? An informal citation or reference is still a citation or reference to something by speaker intent, and is not defined as such by whether or not the audience understands the reference, right?

In any case, citations, references, mentions, and uses of the DK effect are almost always wrong. I’ve seen a lot of them on HN, almost always to imply someone is incompetent, and I’ve yet to see one that’s correct.

Reply on news.ycombinator.com