A year or two ago when LLMs popped on the scene my coworkers would say "Look at how great this is, I can generate test cases".
Now my coworkers are saying "I can still generate test cases! And if I'm _really pacificcccc_, I can get it to generate small functions too!".
It seems to have slowed down considerably, but maybe that's just me.
Eventually, it stops being magic and the thinking changes - and we start to see the pros and cons, and see the gaps.
A lot of people are still in the ‘magic’ phase.
lol
None of us can reliably count the e’s as someone talks to us, either.
a) "know" that they're not able to do it for the reason you've outlined (as in, you can ask about the limitations of LLMs for counting letters in words)
b) still blindly engage with the query and get the wrong answer, with no disclaimer or commentary.
If you asked me how many atoms there are in a chair, I wouldn't just give you a large natural number with no commentary.
A factor might be that they are trained to behave like people who can see letters.
During training they have no ability to not comply, and during inference they have no ability to choose to operate differently than during training.
A pre-prompt or co-prompt that requested they only answer questions about sub-token information if they believed they actually had reason to know the answer, would be a better test.
I think it just points to the fact that LLMs have no "sense of self". They have no real knowledge or understanding of what they know or what they don't know. LLMs will not even reliably play the character of a machine assistant: run them long enough and they will play the character of a human being with a physical body[0]. All this points to the fact that "Claude the LLM" is just the mask that it will produce tokens using at first.
The "count the number of 'r's in strawberry" test seems to just be the easiest/fastest way to watch the mask slip. Just like that, they're mindlessly acting like a human.
It's not reasonable to treat only opinions that you agree with as valid.
Some people don't use LLMs because they are familiar with them.
That is a very natural and efficient way to do it, and also more reliable than using your own experience since you are just a single data point with feelings.
You don't have to drive a car to see where cars were 20 years ago, see where cars are today, and say: "it doesn't look like cars will start flying anytime soon".