Hacker News Reader

Comment by dontreact | Hacker News Reader

My guess is that "making cheaper and faster GPT4" was leading to the rumors that GPT4 was nerfed

I don't care what the official line is, something changed with it. It might be the same model but the way it answers questions and the content of those answers is much more dubious now. I think how it manages its conversation context was gimped in the web chat version. I find it routinely completely forgetting things in longer conversations.

It is the text generation speed that got me suspicious. It seems like ChatGPT's GPT4 suddenly got 2-3 times faster at generating text, while the API GPT4 still generates the text at the same pace as it always has.

I was using it to experiment lazily with machine learning and midi generation via CNN-LSTMs and noticed it happening in the middle of a chat with it. The context wasn't that long and it was providing excellent insights, then randomly started telling me how to make a music game in python and lost all previous context. I tried starting a new chat and adding the back context,had a good bit of code I was providing for context, and it could hardly keep track of the provide code context over the course of a few simple questions/changes, when before it had no problem and would actually surprise with drawing from long past context relevantly. After what ever happened it would randomly assume karras was being used if having not been provided any pytorch specific code for a few messages.

API doesn't have these issue, or didn't a week ago when I was using it often, but something happened with the web model, which I guess is understandable, as the main reason I was using it vs the API was cost savings.

I was thinking how I could optimise for longer conversations and the most obvious thing seemed to be to somehow summarise the conversation ever X messages and ask it to save what it things is important isn’t them.

If they’re doing something like this that would probably explain it.

Copilot censored its response to my question about linked lists yesterday. Ridiculous.

But GPT 3.5 has Turbo but nobody claims it reduced accuracy even though it's much faster.

GPT 4 is now a lot faster and struggles much more. And in theory being faster should be an advantage to an extent, even if the model needs more prodding, as you gain a more seamless back and forth instead of waiting half a minute.

Some people say that people think it's nerfed because when it was slow we associated it with "thinking harder" but I doubt that's the case, and the way to measure that is by loss of ability, i.e. if it is stuck more often as a result of itself and not a bad prompt.

No, my experience is that GPT-3.5-turbo generates markedly poorer results than text-davinci-003 and -002.

Agreed, it's general consensus that 3.5-turbo is a substantially weaker model.

Guess I missed that 3.5 Turbo achieved speed by also downgrading. I didn't hear anyone complain, to be fair I only use GPT 4 because I try to use it to save time and even with 4 you have to verify.

And I guess the trend is that anyone who uses it frequently uses 4 and 3.5 for basic stuff only.

>But GPT 3.5 has Turbo but nobody claims it reduced accuracy even though it's much faster.

Loads of people claim this, I haven't tested it myself though.

Everyone knows that turbo is less accurate.

Turbo is definitely a much weaker model.

More likely to be the non-compete words, I think.

Reply on news.ycombinator.com