Hacker News Reader

Comment by deaton | Hacker News Reader

I think the real disaster is that once you let the LLM work on a project for a bit, you start to lose understanding of what exactly is even happening under the hood in the project. You can take steps to mitigate this, but agents don't exactly encourage the behavior required to maintain a good understanding of what's going on.

View on HN

Like becoming a manager?

When a person becomes a manager, they do or do not have enough time and expertise to review all of the code that they trust the team to produce.

Managers usually get into automated testing; unit tests, integration tests, acceptance tests, and maybe also BDD syntax

Managers and developers are responsible for setting a test coverage threshold for merge approval.

If there is 100% branch coverage test coverage for a codebase, what would coverage-guided fuzzing or property testing find? If there is 100% branch coverage test coverage for a codebase, what is the value of spending resources on formal verification?

How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage?

I've never really understood this argument. If someone's a manager of an incompetent team, no amount of management skill will save the quality of the resulting software. I don't think "just treat LLMs like smart junior developers" fixes this, because well-functioning teams usually also have senior developers to keep things on track. Like, if we handed a team of genius-but-junior developers to the best "people person" manager in the world (i.e. who doesn't actually read the code), would we really expect decent results? Even if the manager tested the code by hand? I honestly don't think so, at least once the software gets past a certain (fairly low) threshold of size/complexity.

If you have a "this is an incompetent team" attitude, that's what quality software you'll get back; whether or not there's LLM use.

If you have a "I trust my competent team to write good enough tests and test coverage isn't my responsibility" attitude, that's what quality software you'll get back.

There are people producing good and excellent quality software with LLMs. Presumably you must discard low-quality code in order to maintain quality.

There's certainly a limit to code quality with current models. On number of lines of code per unit of time, LLM tools certainly already win.

Can costly automated code review for PRs catch most of the problems before they're under consideration for merge?

For example, the vscode repo has extensive copilot integration. Every PR gets auto code reviewed. But with their tokens or the contributors'?

If I take poor quality code (AI-assisted or not) and spend a few hundred dollars on tokens for a next gen model and agent to get to 100% coverage and review for security bugs and CWE common weaknesses, what quality code will I have without refactoring with proven patterns and type annotations and polishing docstrings?

I don't think this argument holds. The OP made the very good point that most quality software teams are composed of both seniors and juniors, with the seniors (and even staff members sometimes) drawing the lines so that the system stays on rails.

If you're a manager now and your "team" is a bunch of coding agents, those agents are hardly junior engineers at best. It is equivalent or even worse than hiring a team of 3-5 junior engineers and letting them run rampant with your code.

> If you have a "I trust my competent team to write good enough tests and test coverage isn't my responsibility" attitude, that's what quality software you'll get back.

As much as this sentiment is nice, it is completely divorced from reality, unless the competence is verifiably there. If you take a bunch of juniors and say "yeah I trust them to do everything well enough," you're going to have a disaster on your hands.

> How does the value of LLM-produced 100% branch coverage compare to no-LLM 100% branch coverage?

This is such a salient question. Sometimes (definitely not always) the test suites produced by LLMs are so trivial it's scary. Coverage can be an illusion for sure.

I have been seeing such trivial shit getting submitted and deleting them from the test suites, but it's disheartening to see that the devs didn't have the better taste to see those LLM "tests" are so trivial they border on noop before submitting them...

What new or existing metric would better assess test coverage than line coverage and branch coverage?

I wrote a tool called tert - I guess it's called an agent harness now - to run various test runners and log test output and coverage output to disk. FWIU stripping spaces from JSON does save tokens. It seems like feeding coverage lines-missing maps into the prompt results in better output, better LLM-authored tests.

"Refactor these tests for maintainability and coverage. Use fixtures, mocks, and parametrization"

>What new or existing metric would better assess test coverage than line coverage and branch coverage?

Substance coverage - testing the actual logic, edge cases, etc. Not mere lines.

You can write tests that cover every line/branch without actually verifying the code does what it’s supposed to. Relying on (unreviewed?) LLM written tests that provide coverage may not be a great idea.

So, a project can have 100% test coverage (the tests run each line or branch at least once without any errors or failures) but fail acceptance tests.

If there is prompt insufficiency, there is probably acceptance test insufficiency.

A more assuming agent could automatically develop a plan that includes presumptive acceptance tests and request feedback before spending tokens

We try to use types as much as possible. And if we can't, then lint rules, etc. Basically, get everything as far left in the SDLC as we can.

And, if/where we need tests, we write the source so they are few, high value, and complementary. Like actual unit tests, not complex with stuff like mocks just to generate trivial coverage.

Formal verification is always more valuable than mere testing, but it's hitherto more expensive. The thing that ultimately matters is closing the loop: how well do the tests match the requirements, both as written and as unwritten in the mind of the customer?

Working with an LLM has given me a real eye opener on unwritten requirements. It's like outsourcing. "Yes, you've given me what I wrote down, but I never expected you do to it in that way"

Causes of failure; Operator skill, prompt insufficiency, model insufficiency, agent insufficiency, insufficient code review, overloaded merge maintainer

> When a person becomes a manager, they do or do not have enough time and expertise to review all of the code that they trust the team to produce.

> Managers usually get into automated testing; unit tests, integration tests, acceptance tests, and maybe also BDD syntax

I can see managers getting involved into acceptance tests, but never in the other type of tests. And the verification mostly is involved into a quick manual testing/watching a demo. Code is not their concern. When there's a bug, they expect you to investigate and fix it.

Given that it's so easy to generate better than test stubs with a model and manually review what it thinks covers, I think it's fair to expect a manager to run and add to the tests.

If devops has done their job, it should be trivial for a manager to contribute to the tests and run the build on git push (or manually re-run the build with the web UI).

If a manager has deploy rights, they should be able to run the tests.

Managers often still have code and team-lead responsibilities IME. First-line managers, that is.

And then sometimes you report a memory leak and it gets fixed by a VP and you wonder if he doesn't have something better to do.

If the memory leak could cost the firm its reputation with important customers, then I doubt the VP had something better to do.

I think what this fails to cover is that managers rely on other people to do that understanding and keeping things in mind. When there's a problem, the manager isn't just saying to a random person "go fix this", most of the time they're relying on somebody who has a deeper understanding of the problem and is not likely to randomly break something else just to fix this one bug.

Fearless refactoring is possible given test coverage? True in my experience; both with higher-risk human-developed software and with agent-coded software.

I haven't yet made myself learn the new swarm of concurrent agents with different specializations/agent_instructions methods yet.

Are multiple worktrees worth the cognitive burden and merge overhead?

A merge maintainer is always in code review mode

Reply on news.ycombinator.com