Hacker News Reader

Comment by amasad | Hacker News Reader

Interesting. This seems like a weakness of natural language understanding. If you rephrase your prompt slightly it would get it right. Try:

  // return even numbers that are also more than 10
  const arrayFilter = (array) =>

It would do the right thing. The fine-tuned version gets your prompt right so maybe it benefited from natural language data. Will look more into it.

View on HN

That's really interesting, indeed I can reproduce this by changing the comment. I also managed to get correct output for this sample by renaming the function.

clearly your original comment was unfair.

Is it, though? The major selling point of coding LLMs is that you can use natural language to describe what you want. If minor changes to wording - the ones that would not make any difference with a human - can result in drastically worse results, that feels problematic for real-world scenarios.

The model is small, so it has weaker semantics.

I get that. But they are explicitly comparing it to Codex themselves.

The criticism stands if you have to continue to rewrite your "prompt" until you can coax out the correct desired output.

I agree. Maybe it interpreted it as return the numbers that are more than 10 in the given array of even numbers.

For example, if the instruction says "return person objects that are at least 20 years old", it might be more reasonable to generate:

array.filter(item => item.age >= 20)

as oppose to

array.filter(item => (item instanceof Person) && (item.age >= 20))

Reply on news.ycombinator.com