Hacker News Reader

Comment by eminence32 | Hacker News Reader

For someone already invested in the Google suite of products (gmail, docs, etc), this sounds pretty useful.

Also, this part seems especially interesting:

> Starting today with responses in English, you can use Bard’s “Google it” button to more easily double-check its answers. When you click on the “G” icon, Bard will read the response and evaluate whether there is content across the web to substantiate it. When a statement can be evaluated, you can click the highlighted phrases and learn more about supporting or contradicting information found by Search.

The biggest problem with all LLMs at the moment is the frequency at which they are wrong (at least when they are used like an internet search to lookup factual info). Any LLM that can improve this (or as in Bard's case, make it easier to detect wrong info) is likely to gain traction.

View on HN

So we can validate Bard using blogspam SEO articles, eventually having those written by Bard or ChatGPT? It's a LLM ouroboros

* the snake eating it's own tail

This feedback loop, when you extend LLM to the horizon, is my primary point against this approach. When 90% of the new training data is from content (a worse version of) it previously generated you get a negative feedback loop to zero quality.

Bard is especially bad right now, at least compared to gpt4. Once they move to Gemini it will be interesting to compare, but until then things like "when does the new Staind album come out" have to be answered the old way with Google search and reading.

I just asked it your question and it seemed to give the correct answer? Also, there's now a built-in button to check its answer against an actual Google search and the UX is pretty slick.

Did it say September 15 or 18? (both answers I've gotten from it over the past month or two). If so, that's wrong. It's September 22. As sibling pointed out though, there are apparently web pages that have it wrong so it's probably not Bard's fault. I also wouldn't be shocked if the date changed and was originally 9/15

It does look like the release date was pushed back a week, about a month ago.

https://web.archive.org/web/20230000000000*/https://staind.l...

https://www.theprp.com/2023/08/22/news/staind-premiere-their... (August 23)

>Citing production delays, Staind have now announced that their new album “Confessions Of The Fallen” will arrive a week later than intended. That full-length outing will now be available on September 22nd.

Fascinating! Thanks that solves what was a pretty big mystery to me, and definitely makes me feel better about Bard

This is what Bard told me: "Staind's new album, Confessions of the Fallen, was released on September 15, 2023." If that's wrong, it's because I also see web pages that have it wrong.

Yep it's wrong, it's coming out on September 22. I don't doubt that some web pages have it wrong though, and it's hard to fault Bard for that.

Yeah, hard to blame Bard for getting that wrong when there's tons of webpages that seem semi-legit that have September 15th. Searching confessions of the fallen "September 15" 2023 yields hundreds of thousands of results which at least for a cached version has that date.

https://allmusicmagazine.com/staind-release-new-single-in-th...

Comparatively, ChatGPT says "I'm sorry, but I do not have access to real-time information, and my knowledge only goes up until September 2021. To find out the release date of a new Staind album, I recommend checking the official Staind website, social media profiles, or reputable music news sources for the most up-to-date information."

So which is more useful, one that doesn't even know there is a new album coming out, or one that knows what was its release date as of just a couple of months ago?

> So which is more useful, one that doesn't even know there is a new album coming out, or one that knows what was its release date as of just a couple of months ago?

To semi-misquote Lewis Carroll: Which is better, a stopped clock or a clock which loses a minute a day? Carroll posits the former, as it is precisely correct twice a day. The trick, of course, is knowing for sure when those two times per day will be.

I'm sure others will feel differently (and I'm very eager to hear from others with different views), but for me there's not that much difference in usefulness between a model that's wrong 5% of the time and one that's wrong 25% of the time. Both models require manual validation 100% of the time.

I certainly feel different. A 5% error rate, to me, is akin to branch prediction, where I occasionally need to rollback but it's infrequent enough that it doesn't pose a significant issue.

However, when it reaches 25%, I find myself meticulously verifying everything beforehand.

Do you feel confident in your ability to detect when the LLM is wrong? For me personally, I don't have the confidence to do this , which is why I feel like I need to verify everything, even in the 5% case.

For me it comes down to "how important is it to be right"? For many of the queries, it's not that important and if I lose the 5% gamble, it's annoying but ultimately inconsequencial. If it's an important thing to be right about, then I will verify it from both.

I don't know how else to ask this: how are you so okay with disregarding accuracy?

What questions are you asking where you don't care if the answer is wrong? I guess I just fundamentally don't understand what the point is. Why not just bookmark the "random article" link on Wikipedia if it doesn't matter anyway?

A significant subset of my questions are basically, 'here's the weird statistics thing I would like to do; what are the academic jargon keywords I should look into?'

All of the LLMs are pretty good at this, and when it hallucinates a response it's totally a decent indicator that there isn't a lot of literature on the subject.

You have to be okay with being wrong regardless. The little snippets after a Google search are sometimes wrong. The blogs or links can be misinformed (For example, most seem to be wrong about Staind's new album release date).

The idea of perfection is silly because it doesn't exist LLM or not. You're not going to get it so it's a matter how often it's right.

I ask a fair number of code questions where bugs and stuff will be caught either immediately or shortly after first use. If 5% of those are wrong, totally fine with me.

I also sometimes ask questions like "what is the tallest mountain in the US" or "what is the hottest desert on Earth" or similar. If I really need to know that the answer is correct, at a minimum it gives me a name to search for to verify height in feet compared to others, etc.

I encounter situations where I used to rely on Stack Overflow, which carries a similar (if not higher) likelihood of being incorrect, often due to outdated information.

For instance, I was recently inquiring about a specific task with CMake and consulted ChatGPT. Initially, the response was inaccurate, but it was obviously so when it didn’t compile. Upon reprompting, I received the correct answer.

If it is writing code, I still have to verify it every time. Then again, that is also true when I write code.

Hopefully my rate of error is way below 25%, so 25% error might not help me that much.

I'd like to think my rate of error is less than 5% too, but that might be close enough that outsourcing the work is still worth it.

Maybe still not worth it if I were that hypothetical programmer that writes lots of code but never writes any bugs.

Reply on news.ycombinator.com