On the one hand, many of the smartest people need significant amounts of time alone to think through their work. On the other, so many decisions an individual in a team makes are constrained by the requirements of the rest of the team. To find these out, you need to coordinate with the team, and that's much easier when everyone is physically together.
In our startup, we deal with this tradeoff with two spatial configurations. We spend part of our time together in an open space where interruptions are common and expected. We spend the rest of our time miles apart, occasionally coordinating over email and IM whenever needed.
Perhaps one way to achieve this in larger teams is to block off large chunks of the week during which it is understood no one interrupts anyone else. Weekends already play this role for many people I know. They often work for a good chunk of the weekend, but are not expected to take calls or return emails. It's when they do some of their deepest thinking and best writing.