On non-compositing windowing systems each window is designated a region of the front buffer to draw directly to it. Drawing outside that region was unrestricted and easy.
The rounded corners in Windows XP weren't hard-coded anywhere, they were just transparent parts of the bitmaps used in the theme engine. Even though they were transparent, they behaved like the opaque parts of the window border.