Hacker News Reader

Comment by simonw | Hacker News Reader

They can. ChatGPT has been able to count characters/words etc flawlessly for a couple of years now if you tell it to "use your Python tool".

View on HN

Fair enough. But why do I have to tell them that, should they not be able to figure it out themselves? If I show a 5-year kid once how to use colour pencils, I won't have to show them each time they want to make a drawing. This is the core weakness of the LLMs - you have to micromanage them so much, that it runs counter to the core promise that is being pushed since 3+ years now.

Specifically for simple character level questions, if LLMs did that automatically, we would be inundated with stories about "AI model caught cheating"

They are stuck in a place where the models are expected to do two things simultaneously. People want them to show the peak of pure AI ability while at the same time be the most useful they can be.

Err too much on the side automatic use of tools and people will claim you're just faking it, fail to use tools sufficiently and people will claim that the AI is incapable of operations that any regular algorithm could do.

Are you sure? Isn´t one aspect of intelligence being able to use, apply and develop tools? Isnt that the core feature that got humanity ahead of other mammals? As an early adopter, I couldn´t have cared less if AI was cheating in terms of strictly academic terms. I care about results. Lets say we´re working on something together and I ask you what is the 123921 multiplied by 1212. As the most natural thing you will dish out your calculator and give me the result. Do I care how you reached it? No, so as long as the result is correct, reliable, repeatable and quick - AND - I did not specifically ask you to perform the calculation specifically by hand or only with your mental faculties. So this is missing from those tools and because we have to remember to tell them for each and every use case HOW to do it, they are not intelligent.

Truth.

The old human vs animal differentiator was humans build and use tools.

Old as in now widely discredited?

https://en.wikipedia.org/wiki/Tool_use_by_non-humans

Now it's effectively a lower bound on intelligence.

If I ask you to count the r’s in strawberry, do you whip out your Python tool?

That depends on the context, obviously. If you had asked me to count them in every "strawberry" in a text file, then I may whip out my Python or some combination of bash, awk and sed. If you asked me in a conversation, I may close my eyes, visualise the string and use my visual cortext tool to count them in-memory. If you gave me a piece of paper with the word on it, I may use my 'eye' or 'finger' tool to count them. There are numerous approaches, based on the problem setting as you see, but one thing in common - you don't need to specifically tell me what tool to use. I will infer it myself, based on the context. Something an LLM almost never does.

This is a very good answer and I'm commenting only to bring more attention to it apart from voting up. Well put!

If you care enough about this you can stick a note in your own custom instructions about it.

If you allow ChatGPT to use its memory feature (I deliberately turn that off) and ask those kinds of questions enough it might even make a note about this itself.

Yeah that sounds obvious, but unfortunately my experience does not align with this (and I've heard from others similar). I am not using ChatGPT, but another tool within an IDE. I was excited about custom or "default" instructions, until it turned out they work maybe 50% of the time. So you end up repeating "make sure to include .github/custom.md" which is effectively the same crap. So we got ourselves a tool which adds to our cognitive load, great :)

Which tool and which model? Those make a significant difference here.

Well, for such a trivial feature, e.g. loading user settings it actually should not matter, as this too, is a problem we solved decades ago in many deterministic ways. But if it does, then we have an extremely fragile technology being promised as solution to all the humanitys problems. The tool we use is Github Copilot with the entire model offering. Out of which we mostly use Claude Sonnet 4. Since over the last several months they started entshittifying it though, as you are probably aware, we reverted from agent mode to mainly using it just as an annoying and verbose replacement for the entshittifed google search.

Reply on news.ycombinator.com