Code Reviews Don't Scale

The problem with AI reviews, is that they add a lot more text to read, the PR description is another wall of text, test steps are very verbose, most comments are just noise.

I believe AI should instead help pointing the dev to the important stuff and help identify noise, this would actually make the review go faster.

since the introduction of AI, the bottleneck for shipping code shifted from producing code to reviewing it, and the PR reviewing experience on Github is lackluster, leaving a lot to be desired.

my team now ships too much code and it sits in review for too long, reviewing AI code is not easy and we need a better tool to review code, especially on large PRs, Github no longer meets my needs or use cases so I had to build something myself.

I got frustrated enough that I built my own reviewer around this idea: instead of adding commentary, it groups files into folders by complexity, surfaces the handful of changes that actually need deep review, and marks the mechanical noise so you can blow through it. On migration PRs it cut my review time down a lot.

As a web dev, I know what works and what doesn't, I hated how AI bloats the PR with more text, so I changed that to make the AI help me categorize the files into folders, splitting them by complexity, and pointing me to the important stuff I need to review indepth, and flag the noise so I can skim it faster, that sped up reviewing time immensely, especially on migration PRs where most is mechanical name changes.

I can share more if people are curious, I'd rather not turn this into an ad.

Consider before AI. In cases where code was written by a trusted human, you had confidence that they were doing the right stuff - the people understood the context and requirements, the changes were based on those requirements, and tests were updated. You could rely on some basic automated checks to run a linter and static analyzer, execute the tests, and check test coverage. You can defer longer-running tests to a later stage, but people can merge their work in and collaborate. You can have blocking reviews for untrusted outside contributors or new employees and review the rest later.

After AI, the input is a mess. I haven't personally seen the worst of it, but I've read about people generating code before any requirements development or analysis. Code is generated without understanding what it is supposed to do, which also prevents good testing. Even with good requirements, people generate solutions that aren't right, and the generation step may remove failing tests or make unintended changes that aren't covered by tests. So much code is generated from untrusted sources and rubber-stamped. Introducing a bottleneck where a human can slow the process down is a good thing here.

Maybe if we can get people to understand how to use AI tools responsibly and have humans take full accountability for the output and changes, we can get back to the old way. Trusted humans and an AI agent assistant can make changes that are solid enough to merge. Independent control over the pipeline and basic automated checks would give just enough confidence in the change, and delayed review before deploying or enabling the feature would work. But not until trusted humans actually maintain oversight over their tools.

You get that with tiny commits, that make confirming the commits' text as easy as possible; if they're hierarchical it's even better, but that requires using a decent commits viewer (absolutely not github and the likes).

I think this is interesting but it seems brittle. It makes me think of the Agile Manifesto "Working software over comprehensive documentation" - not that I evangelize for Agile.