Is this the end of human code review? | Hacker News Reader

Is this the end of human code review?

arxiv.org

1 pointbonjourjoel10 comments

From my reading of this it seems that the bug they were fixing was that there was a chat system which wasn't originally designed to be able to recover from a reload as it didn't track the state of both clients and didn't have any way for clients to catch up. The fix cost $2,430, took 3 days and touched ~50k lines, 189 files (7.1% of the codebase by LoC)

I totally agree that with the right systems we can do away with code review, and it's a solidly impressive achievement by the model/harness. But three days an nearly $2,500 to fix something that is a very foreseeable requirement if you'd done a little bit of planning doesn't seem like a terribly impressive outcome.

I think the more interesting tradeoff here is that the system they had to fix feels like it was "vibed" rather than "engineered" (or maybe the lack of state tracking was a deliberate tradeoff, but it doesn't really read that way) it sounds like they're at the "find out" stage TBH. But maybe spending $2,500 to fix things that would have been easily solved by some forethought is a tradeoff worth making if you're shipping things at ludicrous speed. Maybe you just ship 10 features and only fix the ones that get traction and it works...?

Think we're pretty close to the point where we'll no longer need to prompt AI to refactor or review the refactoring if it's done within tight constraints i.e. absolutely no change to functionality. But when functionality changes may be acceptable as part of it, it's less clear - in my experience AI isn't at a point yet where it can always be trusted to "understand" what the requirements are (including to what extent they can be allowed to evolve), what is or isn't acceptable in terms of user experience etc. That's where human review is still useful and in many cases, essential.

Requirements as product owner are not covered at all by AI, 100% agree.

But in this programming-only example study, the refactor changes the functionnality very seriously, and it has to go deep in the core of the system, find all the traps, plan everything. And the agent succeeds without human code review. Code review was impossible because of the complexity and size of the task.

Code review was not necessary, nor useful, nor essential, and even practically impossible.

This title doesn’t match the article.

How? The agent does something incredibly difficult, and there is no human code review. Why do you think it doesn't match ?

"Here's a case study where it worked once" != "the end of human code review".

Don't get me wrong, this is impressive and capabilities do still seem to be on a generally upward trend with nothing "hitting a wall" despite all the parroting of that phrase, but there's a huge gap between the first time a machine manages something impressive enough to document, and that machine becoming so good at that task that humans need not apply.

Agreed. But also this inequality:

"Is this the end of human code review?" != "the end of human code review"

Now since the problem in this study is more complex than 90 to 99.5% of what a normal ticket is, the question makes sense. The agent does not have knowledge of the project, at the begining of each session, and manages to handle something more complicated than virtually anything a developper has to do. The question holds IMO.

Literally every top 10 LLM can tell you why. Maybe we’re in „this is the end of making other people spell it out for you” era?

LLMs are leaning toward Corporate first order. They will judge anything based on how much corporation would like it said. I call this "corporate RLHF" and they are ALL into it.

The fact is, the agent did something amazingly difficult, that zero human on earth could do. Read the spec if you believe someone could do it. It did it clean, everything is public, and it works perfectly. Zero code review.

https://aisovereignlabs.ai/docs/case-study/liveSession/logs/...

99.9% of the IT dev tickets are easier than this, excluding off topics like discrete math, computer vision algorithms, etc, which are irrelevant here.

Do you think there's anything you've done, with human code review, in the past 30 years, matching the difficulty of this ?

The codebase is a VS Code extension implementing an AI coding agent.

*The main prompt:*

The request was submitted to the agent as follows, in a single natural-language prompt with no accompanying formal specification:

"Analyze this request in detail. Produce a comprehensive report covering understanding and feasibility.

- The system that streams an active session on the frontend within EasyAgents: What is it? A complex system that distinguishes between current and past messages; it is tricky to manage, involves handling streaming pauses, includes mechanisms to prevent data loss due to race conditions, etc. It requires detailed study, it is extremely complex and full of pitfalls, not something to be looked at casually. We will start by analyzing it closely, then extract the code and create technical documentation for it within a new dedicated module (likely located somewhere under`orch/`). "Extract" means moving the code while making it generic, ready for use in both the EasyAgents use case and the standard session use case. The extraction must be done properly, including all associated systems, no sloppy coding. A successful extraction at this stage means the code is separated and reconnected to EasyAgents without EasyAgents actually containing the code itself; the visible functionality remains the same (the change is invisible on screen but real in the codebase).

- What needs to be done? Extract it from EasyAgents, move it, and make it generic.

- What is the goal? It must work with standard AI-code sessions.

- What is the result? Closing a standard AI-code session view no longer kills the orchestrator process; the session remains visible in the activity sidebar.

- Why? We sometimes encounter a bug that turns the view gray (an Electron memory overflow issue in VS Code), which is annoying because nothing is visible. Currently, we have to let it finish blindly until the activity view indicates completion, or kill the request, but that means losing the work done and having to restart the last user message from scratch. With this new system, if the bug occurs, we can close the view and reopen it by clicking on it in the activity list. Reopening it restores the display.

- UX adaptation required: Since the process isn't killed when the view closes, we need a way to kill it. In the activity view, each item currently has two lines (title and elapsed time). Add a third line containing a stop button (stop icon, small size, danger style). This button should appear only when the view is hidden but the process is still running; it (and the third line) should not appear if the process finishes normally or if the view is displayed again. The button becomes disabled after being clicked to prevent accidental double-clicks and bugs. Clicking it kills the process, just as closing the window currently does.

Reply on news.ycombinator.com