I think it refers to the fact that in an immediate mode GUI you adjust the application state while you render the page, meaning that the part of the screen which you rendered before the state was updated shows the old state, while the part of the screen which you rendered after the state was updated shows the new state. The screen is basically "torn" between two or more states and you need to draw another complete frame to bring it back in sync. Not a problem in a game where you redraw everything at 60fps anyway, but not so good for a classical GUI toolkit.
Edit: I guess you could partially solve this problem by double-buffering the application state, so during the first draw you update a state copy, and during the second draw you use the updated state. You won't get any "tearing" but you still need to draw twice.