And even if it did, you can't do clip masks with scanout overlays. So you have to composite (and therefore take the hit of ramping up the 3d capabilities of the GPU).
Yes you can, what you would do is clip against the main framebuffer and leave the window area transparent by inverting the mask. Then you can have the other plane underneath the main one. This is only if your hardware supports that though.
Presumably that would still cause issues at the rounded corners because you need to blend those pixels to make them look nice.