We need to talk about migrations with AI

I've recently did a huge migration in the company, migrating the UI library three major versions, which introduced a ton of breaking changes and noise. the migration is self was relativley easy to do with AI, a task that would've taken men months, done in a couple weeks.

the not so good part about such migrations is that the resulting PR is huge, it does contain some important parts but most of it was noise and prop renames and mechanical changes. the problem is that Github wasn't really helpful with the review proccess, it doesn't provide a good experience for reviewing large PRs (it would sometimes just crash) so it got very stressful to review and keep track of the feedback comments from my teammates and followups

later after that, I gathered the pain points we faced during our review proccess and built a platform optimized for reviewing AI PRs (https://pyor.review). firstly it's a web app and has a desktop app, so it's way more performant when reviewing large PRs, it also include a comment inbox to keep track of feedback, has grouping mechanism so reviewing migration PRs becomes a ton less stressful compared to github, surfacing important files first to review then grouping the noise for easy quick skimming, and it makes use of caching so the content loads faster. There's a ton of other small features it adds that just elevates the experience of code review and we no longer review code on github.

I think that this is one of the better applications for LLMs. Their ability to translate text from one format to another is one of their strong suites. Especially with things like testing libraries, there is immediate feedback on whether the translation was correct.

My company is reaching for AI on much more than this, and of course it's just a personal anecdote, but throwing AI generated code + documentation into our codebase en masse has reduced our understanding of the system as a whole, it's caused a lot of "rubber stamping" on PR reviews, and it's introduced a layer of complexity that we're now going to deal with forever (provided we continue to use AI for everything). Your mileage may vary, my company doesn't adopt what I would consider "good engineering practices".

I'm hopeful that in a few years we can truly settle on the best use cases for AI, rather than considering it an everything tool. Not saying it cannot do a LOT, but from my experience I have found that it just cannot do everything (but we sure are trying).

AI is great at pattern matching many similar-but-oh-so-very-slightly-different cases.

Amazing for systematic refactoring that has a predictable outcome, but would need an ungodly amount of manual labor.