Hacker News Reader

Comment by jiggawatts | Hacker News Reader

Yet another garbage study where “researchers” wasted months of their lives evaluating GPT 3.5 instead of paying $20 to evaluate GPT 4.

It’s like drinking the complementary table water at a restaurant and then leaving a one-star review saying “tasteless and flat”.

I seriously can’t believe this is maybe the tenth such “paper” I’ve seen making headlines.

It brings shame on not only the institution that produced it, but the journalists that failed to notice and published the clickbait headline without even reading the paper.

View on HN

Jesus Christ, dude. To know they used 3.5, you must have read the methods section, in which case you can also see they conducted the data collection in March 2023, the same month GPT-4 was released. Bad timing, I guess, but your rant reduces to telling researchers to just scrap whatever work they did and not bother if an update happens to be released to whatever they were studying before they publish.

It seems like it's only important to bring out a paper to fill a quota somewhere. The researchers probably don't care because no one (e.g. a perspective employer) is going to be actually reading the paper, just the amount of papers they published. That's why linking to something on Arxiv trying to undeniably prove a point ("Look, researchers confirm my point!") most of the time makes me cringe.

Plus, I've mostly given up on journalists. I get clickbait, that's why titles are there for. But clickbait doesn't have to equal badly researched. They could have just as well gotten clicks if they wrote an article "Researchers fail to prove ChatGPT is bad at coding because of this lazy mistake". But they mostly either have no idea of the field they are reporting about or are pushing an agenda. Research? Ain't nobody got time for that.

I'd like to see the stats on how many people use 3.5 over 4 though, I bet you that number is much higher than you think. The price of ChatGPT is being what I'm personally willing to pay for it so if I do use it, it's 3.5 for me.

Maybe the logic is to study the most popular product? Would make sense to me.

Many more people drink tap water than eat dinner at a MICHELIN Starred restaurant, but if you're a food critic writing about the latest fancy restaurant, you had better pay for some food and write about that.

This is especially painful considering the ludicrous pace of advancement in AI. You really have to aim for where the puck will be, not where the puck is. Within a year GPT4 will be out-of-date.

Fundamentally, the entire point of this "research paper" is to compare ChatGPT with Stack Overflow answers, coming to the conclusion that people prefer SO.

Yeah, well, meanwhile Stack Overflow usage has dropped off a cliff since ChatGPT become generally available: https://observablehq.com/@ayhanfuat/the-fall-of-stack-overfl...

I certainly prefer to ask ChatGPT basic coding questions because I get an answer immediately with no argument.

This analogy should be in a gallery, good day to you sir.

But if all that existed at the restaurant was water? The food or better drinks werent released yet.

If that were the case, I would expect to see an explicit reference to the CGPT version in the introduction, and an explanation as to why the authors chose it for the study.

Instead, the only version information appears to be buried in section 3.1.2 ("ChatGPT 3.5 Turbo API is used").

They used 3.5 because that is all that was available at the time.

GPT-4 has gotten so much worse over the last few months, and GPT-3.5 has gotten better.

I still think GPT-4 has an edge but its frustratingly dumb and loses context so often that I'm not sure how much of an edge it really has any more.

It takes time to create and publish a study.

Reply on news.ycombinator.com