I got an even weirder experience. I asked him to calculate the uk annual pension allowance given a specific adjusted income and he correctly cited the current rule, explained it and wrote the correct formula to calculate it.
But somehow he managed to mess up the simple calculation giving a wrong result even if the formula and the number that he used were correct...
It is not designed to do math. It only generates a likely text response. If the exact equation (with exact numbers) did not often appear in its training data then you are very likely to receive a nonsensical answer. It can do some math simply because those patterns have showed up quite often in its training set. This is the same result for anything you ask. It will often imagine things like fake YouTube video links with real-sounding titles, fake function names, etc. I’m sure soon it will be hooked to something like wolfram alpha and we’ll all be terrified at how well it performs at anything math related, since it could likely classify and hand off any of the math stuff to Alpha and then utilize/summarize the results from there in its final response.
Well. Yes? It’s not a computational engine. Al it does is figure out what the probability of the next tokens are and selects the most probable ones