Hacker News Reader

Comment by fosterfriends | Hacker News Reader

I love this train of thought. Code quality is critical, but I don’t think we’re correctly evaling it at the moment. If we could get solid benchmarks measuring the quality of generated code, we might see the models climb those benches fast.

I believe that the era of “ai writes tons of slop code” will be a stepping stone in the longer story, and is simply a current gap in the reward functions.

Per the author - if we can get strong measurements of what good code is, we can train against it and close the gap fast. Excited to see more thinking in this area

View on HN

If you could easily benchmark the quality of code then models would be trained on these benchmarks/metrics.

that is true, but if the metric is what we want optimized, then that's fine.

However it is more likely to be something which can be detached..

Code quality is probably isomorphic to the halting problem, or can be reduced to the halting problem in the simplest case. I.e. it’s intractable.

Sooooo much of what is considered "code quality" today is irrelevant when robots are writing the code. We've been largely optimizing for things like composability/unit testability in the past 15 or so years, and that's primarily a human concern that's unrelated to the final output.

Totally agreed that we're not looking at the correct metric right now. Increasingly, code quality will be determined by outcomes.

> that's primarily a human concern that's unrelated to the final output.

It's an LLM concern too. LLMs seem to do better with well-organized codebases, just like the humans they were trained on.

I don’t agree with this. The things people care about with code quality if you really think about it actually match up surprisingly well with the metric which models are trained to emulate in pre training, namely compression and modularity. Those two ideas actually seem to be universal to intelligent systems. Writing verbose highly coupled code is I think provably stupid, though I don’t know that I could formalize it.

I think Martin F's team tied outcomes to factoring https://martinfowler.com/articles/exploring-gen-ai/refactori... which is crucial to code quality. Even grug brain developer agree.

On the other hand, there are claims that the best languages for robots tend to be terse (allegedly); I tend to regard a rich domain model (static types, not terse) as a major facet of high-quality code.

Reply on news.ycombinator.com