> My biggest pain isn't with the plan. It's with the actual upgrade process. Things _always_ break. Every single time we do a major upgrade, there's this long tail of things we need to fix. Even worse is the undocumented changes that break. This gives me nightmares.
Planning is one way to make those major upgrades go wrong less often. Our idea is that we can break large upgrades down into incremental changes that are individually safe and mixed in to your regular dev cycles, so that when you go to do the large upgrade you're not making such a big change all at once. This means things like fixing every deprecation ahead of time and making sure all the other dependencies you use are compatible with the version you're upgrading to.
We're calling this "upgrade paths" in our tool. Something like a major rails version upgrade is terrifying because of all the moving parts. Often you may have a dozen other gems that need to be upgraded in order to upgrade Rails. You said "If I can incrementally upgrade one package on every PR until I'm up to date, that'd be amazing.". We want to make that the flow for major framework upgrades, so you're merging in incremental improvements in the blocking dependencies ahead of time.
Undocumented breakages are for sure on our roadmap. We have two types in mind today: - Changes that are missing from the changelog - Incompatibilities across gems (for instance there was a time recently where if you had both datadog and newrelic installed and used elsaticsearch upgrading the newrelic gem would break prod)
We have some ideas on how to figure this out automatically (like reading code diffs with GPT rather than just changelogs), but the best way is to see upgrades out in the wild. As we see more and more upgrade experiences from our customers we'll be able to catch these issues and build this dataset.
I'd love to hear other ideas for making major framework upgrades safer.
> * It ran the repo's test suite against progressively newer versions of dependencies. Showing when and where unit tests fail.
Would this be something like `git bisect` for upgrades? We run your CI to figure out where you can upgrade to without something breaking? We're trying to figure this out statically by reading changelogs and building up a database of breaking changes. Our roadmap is to make this database better over time by pulling in more and more undocumented changes (by seeing customer upgrade experiences and by sourcing information from places like github issues). In my experience it's more common (and much more dangerous) that things break when your dependency changes in a way that wouldn't be caught by CI.