The biggest problem is that none of the DEs really fit my tastes — they all have to be poked and prodded into kinda doing what I want them to do, but they're never quite there and it's incredibly frustrating. I've also faced similar issues with starting with a bare WM and snapping together smaller pieces.
To get what I want I'd likely have to build my own DE from scratch, which I'm not even necessarily averse to, but I have no idea where to start with the mess that is X11 and Wayland and all the "build your own WM" tutorials that could be used as a springboard are written for building hyperminimal borderless tiling WMs, which aren't straightforward to adapt for a more "typical" floating WM with titlebars and the like.
So I guess the endpoint of this rant is that it's frustrating that building one's ideal Linux desktop from the ground up isn't all that accessible in reality. The configurability and openness is there on principle but it's difficult to take advantage of past a skin-deep level.