Back then the (usual) hardware didn't support anything; the Xserver just wrote into some 100KByte of video memory, one bit per pixel. The X server had to deal with overlapping, and could.

Only top level application windows were tiled. Dialog windows were floating above them and they were overlapping.

There was no need for hardware support because windows were not buffered, they repainted itself when the system told them to do it.

It was at least partly due to Apple claiming ownership of overlapping windows.

According to "Barbarians led by Bill Gates" this was an intentional choice justified by studies of number of clicks required to perform some tasks.

Great book, btw.