Discussions like that can happen on IRC. @mattm is correct, you're taking the whole team for a few hours (or maybe days) saved, you're on the raw end of the deal. There's nothing saying that type of conversation requires high-bandwidth communication. It's often more efficient to have one programmer waiting on a response in IRC rather than interrupting the entire group.

You're optimizing for the 5% instead of the 95%. I think this is a case of a survivor bias---you remember all of the times having people in the same space saved time and forget all the ideas that went nowhere and interruptions that pulled someone out of the groove.