Jev Based Code Review

My dev branch got broken when I rebased to main after a week of drift.

I had to bisect over around a million commits to the monorepo to find the culprit.

Code is there to be read and understood by the human developers who come later.

The git history is a similar record, that's why the commits that make it to main (the squashed PRs) should cover one(ish) thing each and be self contained.

Something covering 230 files should be a mechanical change like running a linter or the AI is moving an API from one signature to another.

If an LLM generated a 230 file change they are also capable of going back and breaking it up.

One thing they are bad at is comments that are succinct since they almost only ever add words.

Sadly, I know (major) companies that insist it's the process that needs to be solved, because it improves velocity (for some definition of velocity that involves dropping pretty much all quality gates).

Because it's all "AI" now <insert Ancient Aliens meme>

The "moat" is data. And compute. But mostly data.

It is a classifier, that you don't have to train, that seems to work well, and they have made available by API.

One of the reasons for LLM success has not just been that they are "smart" but that they are smart while not requiring you to collect your own data and perform your own training.

There is also the marginal advantage of it not requiring a PhD to know how to use. But most companies can get a competent data scientist on board, so that is not a major problem, they could find someone to train their classifiers. But even a good data scientist can't make data appear out of nowhere. If you can get a pretty good result without needing data and training, for cheap, you're going to take it.

(Let's put aside the fact that almost every company using LLMs these days is irresponsibly NOT comparing the results with their own internal gold standard datasets for validation and calibration. They are just winging it. And Jev allows them to continue to do that.)

SO you do not need to have YOUR data to train classifier. All you need is to benchmark it on your process/data to see that it really is good for your domain (as authors promise) and choose a threshold for acting.

Ahh, and the most important thing you need to do - design well the input context (state), questions with their context (instructions) and output type (because the same question asked as Noul (yes probability) or Choice with yes/no variants give different probability of yes.

I think no one mentioned here, but the obvious difference is that Jev can spit out decisions directly from natural language input. None of these ML models could do that, and other than a full-fledged LLM (which is optimized for conversation and agentic tasks) or some classical NLP models (which underperform compared to LLMs, AFAIK), there is nothing right now that rivals Jev-like models.

Of course it's not perfect, but seems like the right step forward for quick classification/decision tasks based on natural language.

"Plan out the change. Structure the plan so that it can be broken into individual PR's that each covers a single logical part of the change. Aim for less than 500 lines per PR unless the change is trivial and mechanical." (with a lower end model you need to be a lot more precise; with a frontier model you only really need to let it know you care about the PR size)

... in your prompts, coupled with gating PR creation on passing a stringent automated code review that also reviews it on whether or not it adheres to the rules you set regarding complexity of the PR, solves that problem without hiding the complexity.

If you have 230 file changes, that is a process failure unless they're purely obviously committing purely e.g. linter cleanups.

Not quite. Ironically, they get PR-reviewed by someone else's agent. The humans in between are meat-proxies, pressing OK buttons.

That's like telling a structural engineer that they don't need to review their plans.

If you don't like writing code or reviewing it, stop being a programmer, go become a manager or something

In the end... I don't feel better off. More tooling hasn't been the answer. It has only resulted in more complexity, less happiness, and about the same velocity as before we did agenetic engineering.