Even "good" software packages written by people who understand autotools like glib have issues. Just look at the official glib cross compiling instructions: https://developer.gnome.org/glib/stable/glib-cross-compiling... The "correct" way to cross compile it is to figure out all the autoconfigured values yourself, and write them into the configure cache.
Personally, I have had the least trouble with plain Makefile based packages. Yes you often have to reach into the Makefile's guts and modify it to make a working cross build, and you'll have to set the rpath manually, and install the files yourself, but at least it's easier than having to modify an autotools based build.
Autotools "just works", and when it doesn't it's very hard to fix. Makefiles don't "just work", but it's much easier to fix them yourself.