Hardware graphics acceleration started on Unix and was largely used professionally.
Although Apple already started using compositing in 2002 with Quartz Extreme in Mac OS X 10.2, which also had X11 support.
NeXTstep used Display Postscript. AFAIK this has no concept of 3D acceleration or anything akin to it, and was almost entirely unaccelerated.
NeXT slabs and cubes did have a low-function blitter used to composite the main display.
Before that, just well-tuned 68030 burst block copy - they rendered to backing store and did block copies with a special case of vram to vram block copies (used for window compositing).
What do you understand by "display compositing" that doesn't involve 3D hardware?
I bet high-end Unix graphics workstations (CAD, video processing, etc) had it many years before that. The first graphics accelerator was offered by Silicon Graphics in 1983, yes, forty two years ago. It was supported by IRIX, a Unix variant.