Hacker News Reader

Comment by jorvi | Hacker News Reader

As long as these models constantly keep switching up things like the temperatures at which a steak will be medium rare or at which temperature to season cast iron, you will never be able to trust them for cooking. My mother ruined a nice waterfowl for Christmas by listening to Gemini.

And this is inherent to how LLMs work.

View on HN

Not if you let your LLM grounds its truth in established facts. Otherwise they would be useless for programming for example.

I don't understand what this means. I use LLMs daily for my work in programming things, and they regularly will assert things that are not accurate.

A lot like humans, really. People regularly cite something they read, or quote a stat that turns out to be just completely inaccurate. But if you look up the thing, then you have facts again.

Sure, and by the same logic, if I was trying to cook a steak, I would not trust an arbitrary human to know the correct temperature off the top of their head; I'd want someone who I could trust had actual experience with the task I was trying to perform. The difference is that most humans are fully able to recognize whether they've cooked steak often enough to know the correct temperature off the top of their head, and they will say "I don't know" to most random arbitrary questions you ask them outside of their experience. I've yet to see an LLM product aimed at general usage for individuals be willing to say this without someone having to literally direct them to give that as an answer if they're not sure.

His point was that, just like humans, if you ask the LLM to look it up, it will give you the correct answer. Just like humans, if you don't ask them to look it up, you don't know what you're going to get.

And my point is that I've never met a human who confidently asserts incorrect information in such a broad range of domains rather than just admitting that they don't know

The point is that it's trivial to fix the problem of cooking the waterfowl or writing your code with existing tools. Just ask them to verify.

If you have to say "don't make up something" for every possible question you ask in order for it not to make up something, that's a massive usability issue for regular people. If saying "don't make up something" will still result in it making up something up some of the time, that also might be a massive usability issue depending on if "some of the time" means 0.0001% or 1%.

Having a natural language interface where you need to go out of your way to specify that you want an accurate answer rather than just a plausible one defeats the entire purpose of it being a natural language interface for normal people. In certain professional contexts, it can be useful, but I don't buy it at all that it makes sense to ask everyone in their everyday lives to go out of their way to specify that they actually want correct answers to their questions.

> If you have to say "don't make up something" for every possible question you ask

You don't. It goes in the system prompt.

In practice it doesn't though, for any actual products on the market today.

Put it in your own system prompt. They all provide tools for doing this. OpenaAI Custom GPTs, Gemini Gems, Claude Projects, etc. Or you can easily roll your own using their APIs.

When you let the the agent do a test, or tell it to read that doc first, you will ground it in reality. Doesn't mean they are 100% reliable. But without and on their own without access to grounding information, they halluzinate wildly.

The parent comment said that these types of behaviors are inherent to how LLMs work, and your response was to "ground them in facts". From what I can tell, this does not meaningfully change the original point the parent comment was making that the flaw is inherent; throwing a bunch of extra context at it to try to make it happen less often is useful, but it's still just a best-effort mitigation for the behavior, not somehow a way of literally changing the inherent nature of it.

They're only useful for programming because version control, command+Z and "deploying a new version" exists.

Take that away and you'll barely be able to make an app that display a pigeon riding a bicycle (or whatever you ppl are doing these days).

So why isn't this the default mode then?

More expensive.

While yes, their current reliability is too spiky to be relied upon for a lot of things:

Any given failure is not inherent, they are all dependent failures; what is inherent (due to the SOTA in ML, perhaps or perhaps not the architecture) is how many examples they need to get good at stuff.

But how would that be different if she wrecked it following some other online recipe?

Well I wouldn’t trust Gemini with most things.

Really? I've essentially learned how to cook from Gemini. Not a great cook yet, but I can do the basics now.

Reply on news.ycombinator.com