ChatGPT study suggests its LLMs are getting dumber at some tasks

This study was imo completely debunked. https://twitter.com/Si_Boehm/status/1681801371656536068?t=rO...

The claims by that paper are very misleading.

Pretty sick of the posts defending ChatGPT on this one. I have at least 5 prompts I retried that had drastically reduced quality of response and basically forcing me to query a dozen times to get the code it previously just wrote in the very first response. It tries to force you into making skeleton models of your code now if there is any complexity at all whatsoever instead of taking a stab at the code itself. I've done so many retrials of old prompts I've seen it happen enough to notice the actual pattern as to what is going on.

"Where's the proof" Sir I have literally hundreds of conversations, and I am no longer subscribed to ChatGPT4 because of the aforementioned issues, so I cannot really compare anymore. If you were using it in February to write code, try those same prompts now. It's blatant.

If this is true, I have some theories:

- The quality is more random than people would like.

- OpenAI made performance tweaks that caused the quality to drop.

As I've said in previous comments, this seems easily testable by any ChatGPT user. I am curious if this is real, or an example of mass hysteria/delusion.

> Do you use the web app?

> If so, could you please go back in your history and make a new chat with an old prompt which had an excellent response?

> I am curious if you could see the degradation and share the example.

>> u/derwiki

>> To your point, that we haven’t seen widespread examples of prompts that used to work but now don’t, seems telling about the accuracy of claims.

https://news.ycombinator.com/item?id=36798176