Hacker News Reader

Comment by eudamoniac | Hacker News Reader

I find this shocking though not unbelievable. Could you share how you measure this accurately? I'm interested in integrating such measurements into our services as well.

View on HN

We’ve been tracking performance and bugs for years. Including commits those bugs were introduced in. So when LLM-generated code started working its way into our codebases, we have the before and after. And even comparing human generated code today with LLM-generated code today.

I find it unbelievable. I've seen the code LLMs write and it sucks compared to what a typical human produces. The only way an LLM is doing better than human programmers is if your human programmers were producing really terrible work.

What LLM/harness are you using that the results are so terrible?

Believe it or not, most programmers by definition are average. Hence, producing code better than them is not a hard feat to achieve for today's models.

I personally find it unbelievable that you have access to all of the public GitHub projects available and still think the typical developer writes good code.

What exactly do you mean when you say llm generated code? Are people prompting llms for changes and features without reviewing the code or iterating on it and then comparing that to what human writes? Because if so it's not surprising that you're getting worse results. Humans also write code through iteration. You can definitely get llms to write good code by enforcing guardrails and constraints through tooling and agent.md, and iterative reviews to nudge towards what you want. The first pass will look nothing like the committed code. I don't expect the llm to one shot anything.

That must be why literally every single AI-powered code review of completely-human-written work turns up something. >..<

Also, regarding your accusation, you need to contextualize- what language, what use-case, what LLM, etc.?

Reply on news.ycombinator.com