Unfortunately, it seems like these comments are only visible to Librarian's users, which doesn't make it much different from Hypothes.is [1] other than being limited to arXiv, and makes comments less discoverable. arXiv should really add such a feature itself/have open sourced its code so they could've added it there.