Have you tried "explicit template instantiation"? If the library templates have good separation, you can just add a cpp file so the compiler doesn't need to deduce the type each time.
It's not that I'll outright refuse to use a header-only lib, only that I very very strongly prefer not using them.