(34903173/x)+(238 * 2650) - 323326 = 45323434, solve for x
Statistically, no one has ever done this calculation ever before. It's entirely unique.
O1 answered "x = 34,903,173 divided by 45,016,060", which is correct.[1][2]
Now I guess you can pick up the goal post and move it.
[1]https://chatgpt.com/share/6709481a-3144-8004-a7fd-0ccd9e3bc5...
[2]https://www.wolframalpha.com/input?i=%2834903173%2Fx%29%2B%2...
The central problem with math is that you have an infinite amount of space within which to move these goalposts.
How many variants on this trial before we find a mistake?
What is an acceptable error rate?
https://mathstodon.xyz/@tao/113132502735585408
"Here the results were better than previous models, but still slightly disappointing: the new model could work its way to a correct (and well-written) solution if provided a lot of hints and prodding, but did not generate the key conceptual ideas on its own, and did make some non-trivial mistakes. The experience seemed roughly on par with trying to advise a mediocre, but not completely incompetent, (static simulation of a) graduate student. "
How many variants would it take for a human to make a mistake? It's certainly not "infinity", so is this an indication that humans don't reason?
A/B + C*D - E = F, solve for B
It's the same as with coding by the way: it can reshuffle things it has already seen while changing variable names and so on. Ask it something which is not in stackoverflow or geeks4geeks and it goes tits up.
PS: Tested it on GPT 3.5: same answer.