Hacker News Reader

Comment by dvh | Hacker News Reader

I asked: "How many resistors are used in fuzzhugger phantom octave guitar pedal?". It replied 29 resistors and provided a long list. Answer is 2 resistors: https://tagboardeffects.blogspot.com/2013/04/fuzzhugger-phan...

View on HN

> How many resistors are used in fuzzhugger phantom octave guitar pedal?

Weird, as someone not having a database of the web, I wouldn't be able to calculate either result.

I tend to pick things where I think the answer is in the introduction material like exams that test what was taught.

"I don't know" would be perfectly reasonable answer

I feel like theres a time in near future where LLMs will be too cautious to answer any questions they arent sure about, and most of the human effort will go into pleading the LLM to at least try to give an answer, which will almost always be correct anyways.

That would be a great if you could have a setting like temperature 0.0-1.0 (Only answer if you are 100% to guess as much as you like).

It's not going to happen as the user would just leave the platform.

It would be better for most API usage though, as for business doing just a fraction of the job with 100% accuracy is often much preferable than claiming to do 100% but 20% is garbage.

> as someone not having a database of the web, I wouldn't be able to calculate either result

And that's how I know you're not an LLM!

This is just trivia. I would not use it to test computers -- or humans.

Everything is just trivia until you have a use for the answer.

OP provided a we link with the answer, aren't these models supposed to be trained on all of that data?

There is nothing useful you can do with this information. You might as well memorize the phone book.

The model has a certain capacity -- quite limited in this case -- so there is an opportunity cost in learning one thing over another. That's why it is important to train on quality data; things you can build on top of.

What if you are trying to fix one of these things and needed a list of replacement parts?

Not the right problem for this model. Any RAG-backed SLM would do; the important part is being backed by a search engine, like https://google.com/ai

Just because it's in the training data doesn't mean the model can remember it. The parameters total 60 gigabytes, there's only so much trivia that can fit in there so it has to do lossy compression.

It's good way to assess the model with respect to hallucinations though.

I don't think a model should know the answer, but it must be able to know that it doesn't know if you want to use it reliably.

No model is good at this yet. I'd expect the flagships to solve the first.

Where did you try it? I don’t see this model listed in the linked Qwen chat.

Lol I asked it how many rooms I have in my house and it got that wrong. Llms are useless amirite

Maybe it thinks some of those 29 are in series:)

Reply on news.ycombinator.com