Anyway, so it's very useful that everything in X is a window, because it works well with this input grabbing. Also it's nice for borders: the border was one outer window, and the drawing area was a smaller inner window.
I don't quite remember anymore, but I think a bad part about using the X's windows is that the coordinates were limited. Sure you only need 12-bits or so for screen coordinates, but it's convenient if you had 64-bits to handle any possible clipping or relative offset situation.