I think there is a peril in responding immediately (which I tend to do) - you set a weird informal SLA that people then expect you to meet. I've had people call me and say "Hey I emailed you this morning - are you ignoring me?" - they're kind of joking but not really.
It's tough to balance. I agree with you - sometimes, me spending five minutes not only saves someone else an hour, it saves me hours later on when I have to unpick something someone else has done. The skill is trying to identify those critical points without interrupting what you're doing every time you get a mail/IM/whatever. A good team helps with that.