Hacker News Reader

Comment by ilaksh | Hacker News Reader

They did not beat GPT-4. They beat an old measurement. It's still like 16% behind.

Would be happy to hear otherwise.

More specifically, they've beaten an old measurement on one specific benchmark. In my personal experience however GPT-4 is still ways ahead CodeLlama and its derivatives. They are really good for what they are, but you just can't beat the parameter count GPT-4 has and in more complex tasks it does show.

Edit: An example just from a while ago (I know it's not a strictly programming question, but still, c'mon):

>>> Is 10000 bigger than 10050?

> Yes, 10000 is bigger than 10050.

Edit 2: I just asked it to write a simple Python app to render a rotating 3d cube. This is something GPT-4 did almost flawlessly few months ago for me. I used the exact same prompt. I got a program that not only didn't work, but had syntax errors (mismatched parenthesis), referenced variables before assigning them, etc.

Why are people constantly using examples with number comparisons and calculations? That's the one use-case I'd never use an LLM for.

It’s like how people keep discovering that LLMs are bad at letter substitution ciphers. Yes, they are. Because you’re asking it to reason over the inputs and outputs of the embedding model, which inputs it never sees.

Because GPT-4 gets this right. Also please look at my 2nd edit which is a concrete programming example that Phind's model gets wrong on several levels.

> Because GPT-4 gets this right

And a Threadripper CPU can render Crysis, but that doesn't make it a meaningful metric for other processors. Maybe GPT4 happens to get it right in some cases, but in practice it's one of the most inefficient and unreliable ways to do calculations on a computer and is not an intended feature.

Arithmetic logic is *very much* a sought after goal for LLM.

The “intended features” of any sufficiently advanced technology always move forward.

GPT, Bard, etc all want to nail that issue and are dumping cash into it.

Mismatched parenthesis, specifically using )) instead of ) is a common failure mode I observed in all of my codellama testing.

I get the repeating close parenthesis a lot too. Figured it was due to my temperature settings, prompt style or quantization though, but this was my first time running one locally

You are right, they didn't. The reason I said that was because of this post: https://news.ycombinator.com/item?id=37267597 and how people on twitter were talking about it. The goal was for easier recall of the earlier post as this discussion is in continuation of that. The title says that and so does the blog post.

Yes, Paul Graham posted it without questioning, and that's how many people came across it. I attempted to correct him, but I noticed it late (https://twitter.com/LechMazur/status/1695401777863000425).

Yup. I use GPT for unit test generation and Wizard Coder is nowhere near even GPT-3.5 let alone GPT-4. HumanEval performance is not representative of general model capability.

Thanks, that's really all I needed to know. Glad they are trying and no doubt someone will but you have to believe OpenAI has a response in waiting when that happens and they feel the need to deploy it.

Reply on news.ycombinator.com