I was thinking the same thing. Github has some useful keyboard shortcuts available: https://help.github.com/articles/using-keyboard-shortcuts/
Click a line, press y, and it changes to the commit where that line was last changed. Since it shows the tag the commit applies to up top, it's relatively easy to mark what release was affected by a specific breaking change.