I still haven't come across a tool that allows telecommuters to replicate this process.
Most of the good engineers I know are just as adept at hashing out a solution on a whiteboard as they are doing it quietly by themselves, and, given the sample set that I've seen over the years, I've found that the collaborative solutions tends to be better.
* Do you do this every day?
* If so, do you get anything done?
I only being slightly facetious. I agree that there's a lot to be said for brainstorming in person. I'd also argue that there's a lot to be said for brainstorming out of your normal environment (i.e., office). But you don't do this every day. You code every day. Why not rent out some office/conference space for an afternoon every few weeks as you ramp up on your next iteration and save the money on the office space all the while allowing your programmers to be more productive?
We work in an open work space, and I can't even count the number of times that someone has jumped into a discussion with a better idea of how to implement something. If that saves the original people a couple of hours (or days, etc) then we've immediately increased productivity.
I don't buy into the idea that being "more productive" means sitting by yourself in your home office cranking out code.
If you interrupt an office of 8 and one person has an idea that saves 2 hours, you are only breaking even.
I worked in an open space before and it drove me nuts until I finally left. It was fine when there was just a few of us but as they started adding more people, the number of interruptions started to go up exponentially until some days it seemed like I was being interrupted all the time.
I think the happy medium is some flexibility among telecommuting, private offices, group work areas (taking over a conference room, or something less formal), and informal mixing areas. There is no single office environment which is best for all tasks.
I'm at my most productive by telecommuting from home one day a week, and daily from 0600 to 1000 (not always on the main company project); then working in the office, meeting with clients outside, etc. from 1000 to 1800, and then whatever team building, client events, etc. from 1800 to 0000.
For some reason, though, this is a subject on which many people feel qualified to make pronouncements about the wisdom of what other people are doing. I have, for example, had a candidate come in for an interview, look at the office layout, lecture me for 15 minutes about how we must be idiots because we're killing our productivity with our open layout, and then leave because he'd refuse to work in a place like that. At least it saved me the other 45 minutes I would have had to spend interviewing the guy.
Incidentally, no sane software company that I know off is rigorous about enforcing all three in all circumstances. The places I've seen hailed as successes of upper-case-A Agile, on the other hand, are typically gluing together software other people have written (without modifying the software itself, unlike legitimate open source companies) with very little value being added.
Disclaimer: I have nothing against lower-case-a agile, that is developing software iteratively and ensuring a large suite of tests to allow for rapid refactoring; it's the right tool for many (but not all) problems. Upper-case-A Agile (Scrum, XP, etc...) and "TDD", on the other hand, just smell wrong.
Wow, you actually describe the place I worked pretty accurately according to what I witnessed there.
In my (somewhat limited) experience, it's actually "up to 15 minutes", depending on what I was doing. If it's something relatively mindless, I'll be back to work in maybe 30 seconds to 2 minutes; if it's more involved I'll go over my notes (some of which I wrote down immediately at the start of the interruption) and pick up about where I was after 5-10 minutes.
If it takes you 15+ minutes to swap back in after an interruption, you're trying to keep entirely too much state only in your head.
Unless of course you don't think that discussions happen everyday or that it's not ultimately saving us time.
You're optimizing for the 5% instead of the 95%. I think this is a case of a survivor bias---you remember all of the times having people in the same space saved time and forget all the ideas that went nowhere and interruptions that pulled someone out of the groove.
When I've tried collaborative sessions of this sort, they have become prolonged debates over some mediocre solution. Often, a superior solution starts to take shape in my mind, but I'm too distracted by the conversation to develop it. If I bring my unformed idea into the conversation, I'm bombarded with questions that I can't answer yet, and I'm still too distracted to do the actual thinking I would need to answer them. Whatever solution comes out of this, I will inevitably end up improving it dramatically when I have a chance to reflect on it in solitude.
My realtime creative thought process just does not scale to multiple brains.
However, I can certainly collaborate in non-realtime. That is, through email, or some other asynchronous medium that gives me unlimited time for solitary thought.
I don't want to seem like a megalomaniac, but this really is how it works for me, based on a decade of experience. And I don't think I'm an anomaly among programmers, at least not good ones.
The 'oh I need alone time to solve problems' excuse is as awful and trite a cliché as 'programmers drink Mt. Dew and live in the basement'.
Being an in-person employee does not mean you're working in an mega-corporate shackled environment damaging to your fragile hacker self -- it means you are simply able to directly and immediately converse with your coworkers. Excited hackers stammering out 'HOLY CRAP GUYS I FIGURED OUT THE CACHING PROBLEM What do you think about XYZ?' and the ensuing flurry of discussion is not something duplicated via IRC/email.