When I left academia to move cross country, I had a yard sale and at the end of the day a few hundred books were left over.
A young man who had come by earlier still had some interest in the books and so I bequeathed them to him.
Then I moved cross country.
Over five years later, I received a FB message from someone who asked if I had ever taught English in Ohio, and I confirmed that I had.
He replied he wanted to thank me because he was the recipient of those books. He was a Ph.D. student in Math and he read many of the books he got from me and said he learned so much about literature and writing that he never would have as a result of the notes in my margin.
I was humbled that he found my scrawlings worth reading and even more humbled he was able to learn something from them. I honestly think his intelligence was the real key driving his learning, but I am beyond grateful that whatever notes I left in the margins of those books provided enough information to encourage a self-motivated learner to think deeply about the works he was reading.
So while some may find annotations distracting and annoying, there are some that can find those same annotations to be pointers to a fuller understanding of the material so annotated.
EDIT: Change "and" to "to be" in last sentence.