[0] : https://hub.spigotmc.org/stash/projects/SPIGOT/repos/spigot/...
The bad one is trying to follow upstream on each commit, notifying you every time some patch either doesn't work or doesn't merge anymore so you have to fix it manually. This still doesn't stop the cases where the patch in itself works, but doesn't make sense in the project anymore (some code moved around, tests still pass, but the functionality is not reachable). You also need a fair bit of architecture to test each of your patches with on each upstream change.
The worse solution is taking a few days off every time a bigger release is out and porting all patches. It saves on the continuous testing infrastructure, but takes unknown amount of time and you may discover some functionality simply cannot be migrated anymore. You're also likely to introduce many more bugs with rushed development.
I've been through planning something like this on a number of projects before. It's a lot of effort and it should be avoided as much as possible.
I think the product you describe could be actually a really good idea, but a lot of it would have to deal with as clever annotations as possible - what broke, when did it work the last time, who's responsible for porting, what functionality does it provide, is it critical/optional, etc.