Sadly the C++ ABIs are standardized the way that C ABIs are (I'm OK with why but it's unfortunate in that it creates a barrier) so you have to have separate libraries compiled with g++ and clang++ if you use both on your platform (we use both because they catch different bugs, and for that matter exhibit different bugs). But it means you can't simply install, say, fmt in any system-wide directory like /usr/lib or /usr/local/lib
Just as an amusing side note: Common Lisp used to be criticized for the massive size of its libraries and later likewise C++. It was true they were quite large. Now both are criticized for their tiny libraries. Which by today's standards they are.