You will lose up to 24 minutes of full productivity, according to several different pieces of research. For example: "A study by Microsoft researchers tracking the email habits of co-workers found that once their work had been interrupted by an email notification, people took, on average, 24 minutes to return to the suspended task" [1].

The issue is the cognitive load introduced by task switching. You may not realize it on the surface, but breaking away each time to write a reply (or even to a lesser extent just the mere fact that you are now notified there's a message waiting) is causing a huge distraction for your brain, and slowing down your "real" work.

[1] Paul Hemp. 2009. "Death by information overload," Harvard Business Review 87(9).