I hate the way the annotations move around on the right. Mousing over a paragraph makes the current label disappear, and a new label with some extra info slides in from the top of the page. It's incredibly distracting.
I'd look at medium (their "add comment" and "add photo" ctas) - they are clear and not distracting.
Another thing that distracted me other than the animation was how "add tag" got in the way of the annotation link. I'd aim my mouse at the annotation link and the link would move before I click on it. It should stay put.
Would also be great being able to close annotations by clicking outside.
You can actually close the annotations by clicking on the text. We'll be making it possible to close based on clicking anywhere outside or by hitting escape soon.