Hacker News Reader

Comment by freedomben | Hacker News Reader

Bard is especially bad right now, at least compared to gpt4. Once they move to Gemini it will be interesting to compare, but until then things like "when does the new Staind album come out" have to be answered the old way with Google search and reading.

View on HN

I'm sure others will feel differently (and I'm very eager to hear from others with different views), but for me there's not that much difference in usefulness between a model that's wrong 5% of the time and one that's wrong 25% of the time. Both models require manual validation 100% of the time.

I certainly feel different. A 5% error rate, to me, is akin to branch prediction, where I occasionally need to rollback but it's infrequent enough that it doesn't pose a significant issue.

However, when it reaches 25%, I find myself meticulously verifying everything beforehand.

Do you feel confident in your ability to detect when the LLM is wrong? For me personally, I don't have the confidence to do this , which is why I feel like I need to verify everything, even in the 5% case.

For me it comes down to "how important is it to be right"? For many of the queries, it's not that important and if I lose the 5% gamble, it's annoying but ultimately inconsequencial. If it's an important thing to be right about, then I will verify it from both.

I don't know how else to ask this: how are you so okay with disregarding accuracy?

What questions are you asking where you don't care if the answer is wrong? I guess I just fundamentally don't understand what the point is. Why not just bookmark the "random article" link on Wikipedia if it doesn't matter anyway?

I encounter situations where I used to rely on Stack Overflow, which carries a similar (if not higher) likelihood of being incorrect, often due to outdated information.

For instance, I was recently inquiring about a specific task with CMake and consulted ChatGPT. Initially, the response was inaccurate, but it was obviously so when it didn’t compile. Upon reprompting, I received the correct answer.

A significant subset of my questions are basically, 'here's the weird statistics thing I would like to do; what are the academic jargon keywords I should look into?'

All of the LLMs are pretty good at this, and when it hallucinates a response it's totally a decent indicator that there isn't a lot of literature on the subject.

I ask a fair number of code questions where bugs and stuff will be caught either immediately or shortly after first use. If 5% of those are wrong, totally fine with me.

I also sometimes ask questions like "what is the tallest mountain in the US" or "what is the hottest desert on Earth" or similar. If I really need to know that the answer is correct, at a minimum it gives me a name to search for to verify height in feet compared to others, etc.

You have to be okay with being wrong regardless. The little snippets after a Google search are sometimes wrong. The blogs or links can be misinformed (For example, most seem to be wrong about Staind's new album release date).

The idea of perfection is silly because it doesn't exist LLM or not. You're not going to get it so it's a matter how often it's right.

If it is writing code, I still have to verify it every time. Then again, that is also true when I write code.

Hopefully my rate of error is way below 25%, so 25% error might not help me that much.

I'd like to think my rate of error is less than 5% too, but that might be close enough that outsourcing the work is still worth it.

Maybe still not worth it if I were that hypothetical programmer that writes lots of code but never writes any bugs.

This is what Bard told me: "Staind's new album, Confessions of the Fallen, was released on September 15, 2023." If that's wrong, it's because I also see web pages that have it wrong.

Yep it's wrong, it's coming out on September 22. I don't doubt that some web pages have it wrong though, and it's hard to fault Bard for that.

Yeah, hard to blame Bard for getting that wrong when there's tons of webpages that seem semi-legit that have September 15th. Searching confessions of the fallen "September 15" 2023 yields hundreds of thousands of results which at least for a cached version has that date.

https://allmusicmagazine.com/staind-release-new-single-in-th...

Comparatively, ChatGPT says "I'm sorry, but I do not have access to real-time information, and my knowledge only goes up until September 2021. To find out the release date of a new Staind album, I recommend checking the official Staind website, social media profiles, or reputable music news sources for the most up-to-date information."

So which is more useful, one that doesn't even know there is a new album coming out, or one that knows what was its release date as of just a couple of months ago?

> So which is more useful, one that doesn't even know there is a new album coming out, or one that knows what was its release date as of just a couple of months ago?

To semi-misquote Lewis Carroll: Which is better, a stopped clock or a clock which loses a minute a day? Carroll posits the former, as it is precisely correct twice a day. The trick, of course, is knowing for sure when those two times per day will be.

I just asked it your question and it seemed to give the correct answer? Also, there's now a built-in button to check its answer against an actual Google search and the UX is pretty slick.

Did it say September 15 or 18? (both answers I've gotten from it over the past month or two). If so, that's wrong. It's September 22. As sibling pointed out though, there are apparently web pages that have it wrong so it's probably not Bard's fault. I also wouldn't be shocked if the date changed and was originally 9/15

It does look like the release date was pushed back a week, about a month ago.

https://web.archive.org/web/20230000000000*/https://staind.l...

https://www.theprp.com/2023/08/22/news/staind-premiere-their... (August 23)

>Citing production delays, Staind have now announced that their new album “Confessions Of The Fallen” will arrive a week later than intended. That full-length outing will now be available on September 22nd.

Fascinating! Thanks that solves what was a pretty big mystery to me, and definitely makes me feel better about Bard

Reply on news.ycombinator.com