Sure, I can always do something else instead, but that context switching is costly for me as well, when a minute-long conversation, though briefly distracting for a colleague, would have been able to keep me working at the task at hand.
That's one reason why I like the idea of taking forced breaks; they give you a scheduled time to handle brief interruptions, and if you're fairly regular people can tune in to your schedule(e.g. if you're always "working" the first 50 mins of an hour and "resting" the last 10 mins, it's easy to know whether you'd be able to quickly answer a question). Of course, the forced breaks are another form of an interruption from flow; it's still a trade-off.