EDIT: also notable is that the first ChatGPT answer decribes the color as 'orange' but the SVG answer describes it as 'red'.
For example Google Duplex, I remember many being very impressed by this:
https://www.youtube.com/watch?v=D5VN56jQMWM
That entire conversation was just hardcoded with some if-else statements depending on what the other part said. The tech isn't as impressive when you know that, and anyone who interacts with the tech will realize that these are just hardcoded actions very quickly, but the demos will look very cool.
So applying that level of hardcoding to these situations you take a hardcoded script, then put an LLM on top that just generates calls to these hardcoded scripts, and voila now you have the exact same assistant you had before but now it has a more advanced text parsing interface.
But it generates a lot of buzz since people think that the whole thing is AI powered when only parts of it are and the rest wont generalize into typical situations. In fact it is now often worse than before since you can't reliably invoke the scripts you want to invoke, instead you have to fight the AI interface trying to get it to read your question the way you want it to.
You would produce a better and easier to use interface faster and with less work by just exposing a few high level utility functions that are better designed than what ChatGPT generates there. Or at least it would be better to let ChatGPT use those high level functions.
If they could make it solve a maze or find something in a more complex environment they would have shown that.
Definitely makes me think something is up. I wonder if they primed the model with some extra microsoft content to get better demos.
My best guess is that theres an actual example in the training corpus that its pulling from. Or that this is being gamed in some way.
Luckily this problem can be solved as you can prompt the model to not solve math, but instead transform them into a structure that can be fed into an API (like wolfram alpha).
With a multistep process it would be easy to have ChatGPT generate accurate answers to advanced math problems:
1. Instruct ChatGPT to not solve mathematical expression it sees in the prompt, instead only extract the expression(s) into a list. 2. Programmatically scan the output, and make API calls to get the answers. 3. Submit the answers to ChatGPT and tell it to answer the original question with the newly supplied information. 4. Voila! You can solve almost any mathematical question using natural language.