The thing (screenshot and all) was routine done in the 90s with all those fancy non-rectangular applications/demos/launchers [usually available on CDs that came w/ magazines]. They had transparent/alpha zones that copied the screen under them.
Though there was some support for cursor shadow at this point, and the improved GDI heap really helped, making faking a window drop shadow actually pretty feasible. Vista and Aero is the first native support for Windows with alpha channels.
I actually liked Vista. It worked just fine and wasn't unnecessarily reorganized. Plus the improvements to the OS threading model were excellent, which is why 7 is so incredibly rock solid, probably peak Windows.
Often docks will be able to show you the underlying windows behind it, even without compositing. But under the terminal with fake transparency you couldn't see another window behind it, just the backdrop part.
I think X11 had some extensions to allow translucency too for software like oneko, xroach or such.
See https://github.com/onli/simdock/issues/11, and if there really are good alternative solutions I'd be happy for some help.
Still, there was another tech not querying the background pixmaps, but used from old X11 software to do non-rectangular windows. Oclock, maybe.
But alpha, specifically, would be faked at that point. Windows 2000 supported alpha for such things in a basic way, like XP. Vista with Aero and 7 then really expanded the themes and the window compositing.