Hacker News Reader

Comment by nonethewiser | Hacker News Reader

AI should be used for code review but not in CI.

You should already have 2+ developers looking at most PRs. And these developers should absolutely use AI. The PR author should use AI.

But what you should not do is pipe the AI output directly into the PR and tell the PR author to deal with it. That's adding noise to the PR review process. Everything it says is something the PR author needs to validate as relevant, helpful, etc. A human needs to do that before confronting the author with it.

You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?

View on HN

In our new world of non-deterministic output (that's why we love LLMs! they say such helpful/agreeable/sometimes wrong stuff!), I think CI won't be sufficient. CI is in the realm of Quality Control; when I build the thing, is it to spec and does it do what I need it to do?

But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.

In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time. For example, if a hotel chatbot starts telling guests checkout time is at 11pm instead of 11am, that's a real operational problem and ideally should be caught before it impacts customers.

LLMs introduce new classes of problems/risks that we are just starting to understand and develop the tools to manage.

>But when the model can shift underneath you, I think it will put pressure on Quality Assurance which is an evergreen task. As a parallel, drug manufacturers don't just test their molecule and manufacturing when they build it, they test it regularly to ensure defects haven't crept in because of some unexpected input to their final output. I think that is similar to how software will evolve.

The instinct seems good because it's impossible to keep up with all the details if you are running AI full-blast. Absolutely impossible. So testing outputs makes sense.

I have a hard time seeing exactly how we get from here to there. But intuitively I would not be surprised. One of these thing where quality may drop 20% but you can scale 100x.

> In my work with LLM-included software, I built a tool that evaluates text output relative to a baseline of what's expected. It helps to ensure things don't drift over time.

Is that hotel example real? Curious how exactly you employ this technique—my naive idea was, if talking software development, a sort of 'sanity-check auto-linter agent' catch errors on a regular basis (every 10 seconds, every write, w/e).

Well, it's a real example! No, it's not a true story that I'm aware of, but there are plenty of examples of real chatbots run amok.

I refactored the tool out of my application and it's available now at https://endpointevaluator.com . There's nothing there someone couldn't build themselves, but then you have to spend that time building and maintaining it. I think the ongoing and long term cost of maintaining all this LLM-generated software is under-appreciated. So hopefully there is still space for outsourcing tools that are generic enough to be used by many and provide good value for cost.

If that aspect is critical to your workflow, it seems like you could run the model of your choice off of hugging face, on GPU hardware under your control, so the model won't shift out from under you.

That's one good way to reduce the risk, but I don't think it eliminates it.

Even with the same model and the same input, the output is inconsistent. And what I've observed is that as the size of the input and output grows, the consistency and accuracy of the output seems to decrease. It gets more complicated when you don't control the full input, such as a chatbot with customers.

I think the problem remains even if it can be mitigated by freezing the model and the hardware, which carries the tradeoff of requiring a model you can download and run on your own so you can't use the SOTA models.

Our agents automatically review our PR's - the authors agents automatically see the feedback and make fixes, and automatically merge when everything is green.

A well authored CI review process is significantly better than any human could do. We have the AI review not only the changes but clone and investigate all related repositories that integrate with the code to evaluate interactions, check all open and historical issues that may be related, review production/development logs for the feature, etc.

It will then also go update issues/tasks to reflect the changes after the merge.

Similarly you can have automated agents watching post-deployment to mitigate risk in the "a new feature just went out" time-frame. Just having someone check the related logs in the hours after deployment can catch a lot of issues that otherwise would turn into a fire-drill.

Must be a token burner setup. Nice if you really got it to work. Now also add business agents that come up with new features based on actual research. Your automated humanless company is almost there.

Sounds like an AI-OS slop machine…been there, done that!

They're making money, which is more than most ICs living in cost centers can say.

The human code monkey sometimes thinks too highly of the task of coding. That's never once been the job to be done.

Every engineer is going to become a master of understanding and measuring need, opportunity cost, organizational support and headwinds, political will, etc. These are the true features of the job. The coding part will fall away entirely.

Deliver value. Code used to be the expensive part, but now it's becoming cheap, fungible, and ephemeral.

This all assumes LLMs can maintain a steady state of code while extending it.

JFC, get off the Kool-Aid dude ;)

...of course you're likely just trolling, hard to say with the drivel coming out of Silicon Valley the last few years. In that case: my apologies and keep up the good work :D

Should this not all happen before the PR is created?

My question as well. Why is this happening in ci/ci? Why aren't contributors performing the automated review and correction step locally?

The only answer I can imagine is that it supports reviewing patches from people who aren't using AI or review by humans but parent doesn't mention this.

What if your working on work that isn't easily worktreeable due to available resources but you want to work on multiple things? Seems pretty normal to leverage CI as a cloud compute resource instead of clogging up the machine i use to interact with my agents and communicate with others.

Yes, it should. Reduces a lot of noise.

This is where we landed too. It works remarkably well. AI guidance is really the crucial thing here. I know a lot of people don't like this, think it's really bad, think it just results in slop... but it doesn't or at least never results in dead end slop. If you find what you get out of AI is weak in some area, then get it to focus on making sure that it does a better job, give it more guardrails / guidance etc till you are happy with the result.

> You wouldn't ask an agent to review a PR then just copy/paste the output int PR would you?

Hasn't everyone already got agents directly adding themselves to PRs and leaving comments (occasionally useful)?

They shouldn't.

I personally would never dump a claude code response into a PR body an ask someone to address it. I can't believe any developer would find this to be appropriate or fulfilling the duty of reviewing code.

I think it's pretty clear that what you should do instead is go over the result and communicate to the PR author anything you think should be addressed. You don't just say "here are 8 things you might want to address but I cant actually speak to any of them."

Indeed.

To add to this, the fact that Claude so often tries to deviate from defined architecture can be really frustrating if you're working in a mature codebase.

I think there is a subtle difference between you-or-I dumping a chat paste into the PR, versus an automated agent directly commenting in a PR. The former carries a human social implication, the latter does not. The weight of them is different.

That scenario would be annoying but it's not what we have. The agent joins the PR leaves a few comments and responds (with mixed success) to comments. Personally I'd go back to 2020 and never see an LLM again quite happily but that genie is out of the bottle.

Unfortunately, the results speak for themselves and I could probably find dozens of edge cases which the AI code review caught which would have otherwise been missed, probably until production.

Yes. I sympathize with the “every PR needs human eyes on it” ethos but fundamentally the only thing that scales with AI PR generation is AI PR review.

If you are talking about AI's opening PR's then maybe you have a point. But a person opening a PR should not be met with unvetted feedback.

To be absolutely clear, AI should be used for PR review. It should be used many times. By the PR author and by all the reviewers. It should not just be piped directly from the agent to the author inside the PR. That causes the author to triage every comment.

That's probably true. And it is also a glaring red flag saying "STOP LETTING AN LLM GENERATE PRs". Far too few will heed that red flag, I fear.

I don't get the point of AI pr review.

If the AI can review the pr, it can do so before the PR exists in the first place.

It's a tool/skill/prompt. Just share it with the end user or encode it in the codebase in some files.

It may actually be even more useful as that person will use it often and improve it or can it.

That sounds good. But it doesn't work in practice from my experience.

The exact same skill in the hands of one person will have vastly different outcomes in the hands of a different person. The review skill I built myself has been shared with folks. They (say they) use it before they put up a PR. I still catch things with the same skill. That is evolving as I catch the model(s) at BSing. Even if I would re-share it all the time, when I catch it, these other people using the same skill wouldn't catch it.

And that is exactly why building a general purpose review agent (or skill) is harder than me having my own evolving skill. If I have to "properly" release a skill/some automation, I will likely err on the side of not having too many false positives. That is harder than still relying on a human to review the AI reviewer. The same people that were really good actual human reviewers of a PR are probably the ones that are good doing the same thing while AI assisted. And the ones that really needed these other human reviewers and processes to help them be productive are probably going to produce a vibe coded mess if left to their own devices with nothing but some AI.

Have you tried running the same review skill back to back in fresh sessions? My experience has been: 1) build complicated thing with LLM, 2) run code review, 3) it finds like 8 things, 4) fix them, 5) run another code review, 6) it finds another new 8 things, rinse and repeat.

I'd guess people truly are finding things pre-review. It's just that LLM review seems to have a limit to the number of problems it can or will find. While at the same time, LLM-written code can be an almost limitless source of bugs and errors.

It has no sense of when an architecture is doomed and needs fundamental changes. Instead, it will happily continue to chase individual bugs nearly to infinity.

But, maybe your review skill is better than the ones I've tried.

With the way memory systems work, I can see the value in having a different person's AI conduct the review as that AI's 'memory' is going to have a slightly different perspective aligned with the developer piloting it

I'm glad I work in places where there's no such silly pointless rules like how many people need to review a PR.

The PR author asks for feedback if it needs feedback, otherwise it merges it, period.

I don't know why and when the world got convinced that all this bureaucracy is a "best practice", when it's just a practice, that can be good, or a waste of time depending on many factors.

Like do doctors when analyzing your medical records are like "I have a diagnosis, but first let me have two colleagues review it"? No, unless they are at the beginning and it's part of the process or they need one.

And that's way more serious than the crap most HN is writing.

My partner gave birth last month and the hospital had a system where every hour a second midwife would come into the room and look at all the charts and review their colleague’s work with a second, fresh pair of eyes.

Double-checking in medicine is a practice that is usually used only for specific high-risk scenarios.

Maybe yours was, albeit it sounds strange because hourly it doesn't even happen in intensive maternity-unity cares where there's a high risk for the child's health.

In my experience with the Claude Github integration, I found it to be pretty helpful. It’s had a pretty good success rate of catching bugs before they get to master, and for simple ones I can ask it to fix itself.

> you should have 2+ developers looking at most PRs

It’d be nice, but usually not the case in my experience. More eyes is better. AI review should not replace human review, it should supplement it. I find myself spending more time doing end-user testing instead of looking at code vs before.

Did you miss the part where I agreed AI should review PRs?

We had a two human PR requirement until recently we dropped it. It was slowing us down too much now the human developer creating the future is obviously writing it all with AI so they need to check it then depending on the feature and it’s use it requires a PR but it’s not universal and we’ve stepped up our automated test Tan X what it used to be it’s been so far fewer bugs better delivery

I, too, have noticed a degradation in my grammar when I converse too frequently with AI.

speaking or typing?

If you imagine someone speaking that comment out loud, but speaking as if they were giving a keynote at a Meta or Apple dev con, it becomes much easier to read. The commas and dramatic ellipses just fell into place as I read. Like the matrix, but instead of green kanji raining down, it’s readability-increasing punctuation. lol

But is the grammar load bearing?

Are people getting burned out?

Getting burnout from LLM-assisted coding or any other activity is here.

What exhausts and leads to burnout, is probability.

Sometimes you get so much dopamine if it goes right.

And sometimes you feel like an idiot because LLM does so many mistakes even if you think that you prompt it right.

As the pace is so fast, you change your mind’s state so frequently and quickly that it may lead to a (mild) burnout.

> As the pace is so fast, you change your mind’s state so frequently and quickly that it may lead to a (mild) burnout.

This is so true. Cognitive brute-force always when dealing with AI.

Certainly not always. There's a hedonic adjustment which happens however, where some tasks go very smoothly without much specification and a lot of "you know what I mean" to the LLM, while others then require you to get painfully specific after it badly misinterprets your intent.

Or maybe you can just get too spoiled with it grokking your intent, then become so vague that your vague ideas are actually just bad ideas. Certainly has happened to me.

After almost 4 years with LLMs, if my prompt is too vague that I don’t know how to ask precisely, I use this prompt “I have this idea… {description here} How would ideal prompt look like to make idea realize?” And in second round I polish prompt myself. It usually works.

> You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?

I would if the PR was clearly written by AI. I'm fine with the PR author using AI, but only to draft the PR. They should be editing the shit out of it for the final version before submitting it.

Workflows:

- Human -> AI: OK

- AI -> AI: OK

- AI -> Human: Not OK (at least here)

AI code reviews are the same as AI pull requests. Do you want that firehose? The suggestions might be good. But do you want to add a deluge of work items to that part of the workflow? Do you want the PR owner to be the one to triage feedback before it gets to them? After they have already done it themselves, perhaps with an even better model?

I would and do.

works perfect

https://github.com/dzmitry-lahoda/dz/tree/main/agents/skills...

burns half of day sub of astra for 200 USD. runs 1 hour on our repo.

finds bugs missed by coderabitai, devin-integration-bot, codex and copilot (I ask them first until nothing found, yet my orchestration finds more).

I do not hardcode our application in orchestration, but run subagent for applicationdomainproduct detection.

also I use agy 200usd sub for second-opinion as one of steps for false positive elimination.

Human reviews are going away for most PRs these days. If the LLM decides the risk level of the change is low, the PR can be merged without another human in the loop. That's at least the direction many companies are taking. Only require a human judgement when necessary.

We’ve added AI to our auto review process. It does expose when the author is not confident in their solution to push back. Which is interesting. But we do try to target specifically at our patterns and safety.

> You wouldn't ask an agent to review a PR then just copy/paste the output into the PR, would you?

of course not, it's disrespectful to the author even if they used an LLM to generate the code. what you should do is actually try to understand what the LLM is saying about the proposed changes, check whether it's talking shit or legit, and if legit, rephrase in your own words why you think a certain thing should be changed

Nah, we have AI code review at Google and it is shockingly good at catching bugs no one would have noticed. I absolutely depend on it now.

Curious, which kind of model is used ? do y'all prefer using a lightweight model or a more capable one for code review ?

It's almost certainly Gemini but I don't know if it's Flash or Pro.

I would assume though advanced reasoning models would be best, especially since it's out of band. If I were building it maybe I'd run two adversarial models.

would you share a bit about the style of the code reviews and how its initiated? im trying to build something similar for my company

I have no idea how it works internally. Your assumptions about its workings are as good as mine.

It leaves comments though like a human reviewer would. I assume the key is a good prompt making sure the comments aren't too frivolous.

2+ devs looking at PRs? U have never seen that in 20+ years.

Echoing phrasing from the bef-ai-re times: this is the way

We totally use it in CI. We were thoughtful in how we built our review bot and it catches a lot of issues before humans lay eyes on the code. Typically they are lower level code issues that humans would not have noticed. It's not the comprehensive review solution, which is why we still require some level of human attendance to the code, but it's very much worth using. We do set limits on PR size and description quality as well to mitigate the harsher AI slop issues.

Notice you haven't even attempted to address why it's important to have in CI. We already agree AI PR reviews are good. But why should you add this new responsibility for the PR author of triaging feedback?

> Typically they are lower level code issues that humans would not have noticed

The author does not triage the feedback. Review bot assesses priority and agents fix the issues the automated review discovers. Nothing is blocked by the bot, humans can ignore if they think the feedback isn't helpful.

We use AI heavily in development but everyone has their own setup and way of approaching use. AI in PRs provides a consistent review layer beyond what the engineers do themselves, and catches a previously undiscovered issue in about 75% of the PRs.

so I assembled orchestration which catches design issues, multilayer livenesssoundnesssecurity issues.

kind of high level.

did you noticed that astra started to write better comments which look as it understands something?

Haven't had a chance to test astra vs 5.6 in these reviews yet. I've seen mixed feedback on astra generally so have hesitated in the upgrade. Have you tested and found any difference?

Reply on news.ycombinator.com