Comment by M4v3R | Hacker News Reader

Comment by M4v3R | Hacker News Reader

Parent Full thread

More specifically, they've beaten an old measurement on one specific benchmark. In my personal experience however GPT-4 is still ways ahead CodeLlama and its derivatives. They are really good for what they are, but you just can't beat the parameter count GPT-4 has and in more complex tasks it does show.

Edit: An example just from a while ago (I know it's not a strictly programming question, but still, c'mon):

>>> Is 10000 bigger than 10050?

> Yes, 10000 is bigger than 10050.

Edit 2: I just asked it to write a simple Python app to render a rotating 3d cube. This is something GPT-4 did almost flawlessly few months ago for me. I used the exact same prompt. I got a program that not only didn't work, but had syntax errors (mismatched parenthesis), referenced variables before assigning them, etc.

Why are people constantly using examples with number comparisons and calculations? That's the one use-case I'd never use an LLM for.

It’s like how people keep discovering that LLMs are bad at letter substitution ciphers. Yes, they are. Because you’re asking it to reason over the inputs and outputs of the embedding model, which inputs it never sees.

Because GPT-4 gets this right. Also please look at my 2nd edit which is a concrete programming example that Phind's model gets wrong on several levels.

> Because GPT-4 gets this right

And a Threadripper CPU can render Crysis, but that doesn't make it a meaningful metric for other processors. Maybe GPT4 happens to get it right in some cases, but in practice it's one of the most inefficient and unreliable ways to do calculations on a computer and is not an intended feature.

Arithmetic logic is *very much* a sought after goal for LLM.

The “intended features” of any sufficiently advanced technology always move forward.

GPT, Bard, etc all want to nail that issue and are dumping cash into it.

Mismatched parenthesis, specifically using )) instead of ) is a common failure mode I observed in all of my codellama testing.

I get the repeating close parenthesis a lot too. Figured it was due to my temperature settings, prompt style or quantization though, but this was my first time running one locally

Reply on news.ycombinator.com