We all know the “how many Rs in strawberry” but even at the word level, it’s simple to throw them off. I asked ChatGPT the following question:
> How many times does the word “blue” appear in the following sentence: “The sky was blue and my blue was blue.”
And it said 4.
But that is a totally different problem from “rate how red each of these fruits are on a scale of 1 (not red) to 5 (very red): tangerine, lemon, raspberry, lime”.
LLMs get used to score LLM responses for evals at scales and it works great. Each individual answer is fallible (like humans), but aggregate scores track desired outcomes.
It’s a mistake to get hung up on the meta issue if counting tokens rather than the semantic layer. Might as well ask a human what percent of your test sentence is mainly over 700hz, and then declare humans can’t hear language.
Attach a probability for the answer you give for this e.g. (Answer: x , Probability: x%)
Question: How many times does the word “blue” appear in the following sentence: “The sky was blue and my blue was blue.”
```
Quite accurate with this prompt that makes it attach a probability, probably even more accurate if the probability is prompted first.
But LLMs can write code. Which also means they can write code to perform a statistical analysis.