ChatGPT went from correctly answering Math Problem 98% of the time to just 2%

I'm not sure ChatGPT is fundamentally good at these type of problems. Probably not. One of the examples I remember seeing was "is X a prime number" which ChatGPT can't really reason about it, but it could remember if it saw it around

Drift analysis are certainly interesting, but you need to know what kind of questions and examples are being fed there

Wolphram Alpha is just better at answering those

"Similarly varying results happened when the researchers asked the models to write code and to do a visual reasoning test that asked the technology to predict the next figure in a pattern."

With the ability to chain to the wolfram alpha plugin for math, the above blurb is a little more concerning to me.

https://archive.is/DtcGz

<Full Title> Over just a few months, ChatGPT went from correctly answering a simple math problem 98% of the time to just 2%, study finds

Is it government pressure to dumb down the model?