Hacker News Reader

Claude's code review skill, in particular, can find some good stuff. But it has some big blind spots around certain types of code. And it likes to come up with a lot of nits too—I think it's really really trained to try to always find between 2 and 8 things or somesuch. Good news is that it is very receptive to "nah" on the bikeshed ones and doesn't stick with them, but will stick with big issues. It'll probably bring up a few more nits though that it didn't bring up the first time!

But I can completely believe that someone who knows the code by heart would have a better signal to noise ratio on their reviews.

I'm trying to find the sweet spot because I've found some NASTY bugs Claude missed, and also had Claude find some nasty ones for me. And this is in codebases with tens-of-thousands of AI-generated lines of code + AI-driven reviews. So I want to bring both to the table.

The existence of some of these major "oh man that changes a lot of our assumptions" bugs that were only found because someone poked on the agent and said "I don't think you're paying enough attention to this" justifies that, IME.

And the better you are at pointing the agent at the truly-important parts, the better the agent's gonna be at finding shit you missed.

Claude's code review is a lot less interesting then getting Claude to reproduce the bugs it claims to find in code review, which has had an absurdly high hit rate for me.

The biggest problem I see with how a bunch of people use these tools is they go to them as an oracle, rather then letting them be plugged into and interactive with problem.

And it's in that later context that Claude is amazing: it can run tests and setup scenarios which would take days or get stuck in some weird problem loop. And then you can just say "okay, walk me through this problem" and see it yourself right there.

> The biggest problem I see with how a bunch of people use these tools is they go to them as an oracle, rather then letting them be plugged into and interactive with problem.

At work they have some scale of how "advanced" of an "AI engineer" you are.

IIRC, using the model interactively means you're stuck and "level 3." IIRC, level 5 (the best) is having some agent interview you about what to do, generate a story from that, then some other agent consumes the story and implements it, etc. I think you're supposed to check their work at each step, but that sounds inhuman and unfulfilling.

A lot of bugs are essentially that the code does something plausible correctly, but it's the wrong thing to do. People who are familiar with the codebase and its purpose can often spot those bugs. Those with less experience usually can't.

The issue is tacit knowledge, or implicit context. Most details are never written down. If you don't know them from experience, you have to guess. If you guess, you often guess wrong. And even if you know something from experience, you are often not aware of it, until you see something that violates it. So it's not possible for you to write it down in advance.

You can replace people with AI in the above, and nothing fundamentally changes.

> This, I find completely unbelievable.

You both have anecdotes. Anecdotes don’t “cancel” each other out.

Here’s a third one. In some of the code reviews I’ve encountered that AI gives a lot of feedback, it’s just providing noise. Things that should be ignored or when following the feedback causes more harm which requires more token to “fix” later on. That can also happen. Sometimes the thing it spits out goes against the common sense, and sometimes it works very well.

> So, to anyone who insists on trying to keep up with AI, I say: good luck.

This I agree with, for a different reason. It’s like trying to swim in a sea of honey and trash mix. It’s exhausting.

> Here’s a third one. In some of the code reviews I’ve encountered that AI gives a lot of feedback, it’s just providing noise

Something I've found fun is seeing how long it takes for an llm review tool to come back satisfied with a PR. Think 100 lines of code changed, nothing terribly significant, but also not trivial. I'll have a local Claude session setup to babysit the PR and wait for feedback, accept all the recommendations, push the change up and request a review. I cap the number of iterations at 10 just so I'm not blowing a stupid amount of money. I've yet to come up with a PR where the llm reviewer is satisfied with the changes and has _no feedback_.

So where's the reasonable cutoff point for llm based reviews?

Huh, I've had many times when `codex /review` comes back satisfied on the first shot, both with handwritten and LLM-assisted PRs.

We use only one round of LLM review, and have discussed as a team still assessing recommendations, not just accepting everything blindly. So somewhere in the (0,1] rounds of review. Sounds like our preferred ratio of human to LLM involvement is different than yours, though.

>> Here’s a third one. In some of the code reviews I’ve encountered that AI gives a lot of feedback, it’s just providing noise.

This is completely normal if you haven't written your own custom skill with instructions on what types of issues the AI should emphasize, which ones would be considered nits and which ones aren't a problem at all.

In our repos we use a classification system: blocker, should-fix and nit. Each one has specific definitions, criteria and examples encoded in the skill file. When the time comes to review a PR, agents invoke the skill, and frankly do a stellar job. A human then reads each finding, asks the AI follow-up questions and makes the final decision in terms of whether the finding goes in the PR review.

The reason I know this works is that we have one guy on the team who does not use this skill, and blindly throws his agents at PRs. And the results are exactly as you describe.

> This is completely normal if you haven't written your own custom skill with instructions on what types of issues the AI should emphasize, which ones would be considered nits and which ones aren't a problem at all.

"You're holding it wrong" :)

> "You're holding it wrong" :)

It's a general purpose tool. There are different ways of applying it, with different results.

> This is completely normal if you haven't written your own custom skill with instructions

This line of thinking comes from the assumption that LLM are somehow infallible and can’t be wrong, which is obviously not the case.

Writing skills and other magic incantations is the first thing that comes to mind of anyone who sees the review results for the first few times. What makes you think we didn’t do that?

As someone who owns a company:

Some holdouts excel at what they do to the point of retaining significant value, earnestly worried about being out of touch with what we do, some holdouts just suck and are bitter.