Not sure I follow. I don't take time off to "check" my email. When I get an email, I get a notification. I open it up, respond, and get back to what I was doing. Where's the time lost in this?
The issue is the cognitive load introduced by task switching. You may not realize it on the surface, but breaking away each time to write a reply (or even to a lesser extent just the mere fact that you are now notified there's a message waiting) is causing a huge distraction for your brain, and slowing down your "real" work.
[1] Paul Hemp. 2009. "Death by information overload," Harvard Business Review 87(9).
If you get interrupted a lot it helps to take notes on a pad, kind of use it as your "stack" and just list out what you're doing so when you're done you can go back to that frame of mind more easily.
Plus composing a reply is time-consuming in itself- you often need to look up something, check a document, etc. etc.