This has also been my experience with home-manager. The promise is great - it's like DLC for Nix that just Does More™. Then you get into it, and realize the modules are a lot more fragile and most of them are outdated. Then you're a few hundred lines in with workarounds, DBUS declarations and all sorts of declarative ricing and go "is it worth it?".
Since then I just removed all the home-manager stuff and relied on NixOS' native config. Things are much more stable and easier to define, and I haven't found a whole lot of things that are truly undoable without it. It's slower and non-declarative, sure, but my sanity is all the better for it. I hope I'm not the only one out there.