The comments about library etiquette are on to something.
But I think it also helps that most of the other people in the library are probably strangers, and if a couple of them are talking it's probably about something quite different from what you're working on.
"Team" conversations that might possibly be relevant to your stuff are the worst distractions.