In fact, it's especially bad when it looks like I'm doing nothing. Often, I'll be working out the structure of a long article in my head, crafting the argument and how each paragraph contributes to it. The whole thing collapses when I'm interrupted. At that point, asking if I want a cup of tea is likely to receive a sharp response.
Today, I don't bother explaining: I just make sure that I have a place I can go where I won't be interrupted.