Fragmentation is not
the problem. The future of C++ is to have metadata exchange, at least the way I see it.
This means that packages can be compiled with any build system, the one that fits you better. In the event that you cannot find a binary that fits you, your package manager can compile on the fly. There are also solutions to keep binary artifacts in Conan. This means that most of the time you can consume pre-made packages in vcpkg/Conan for casual consumption but you can still have binary repositories with fully customized builds for all your permutations of compilers and systems and debug/release. Do not forget this is a hard problem, we are talking to compilation to native, sometimes you need full speed and a custom compilation, not about portable Java bytecode.
But you can still set up and tweak recipes in Conan and upload to your binary repository (what we do at my company).
However, none of these things tie you to a build system, since conan can generate .pc, .cmake, MSBuild, XCode and way more metadata to consume those packages.
You can scale from simple to fully customized. For example before we had a libcxx that we compiled ourselves and pointed all deps and patched packages and could build something with like twenty-some dependencies with that customized libcxx optimized for us. Can you do that with Cargo? Serious question, I did not make extensive use of it.
However for simple uses you can just drop a conanfile.txt and conan install your profile and start to consume your packages. All of this works with CMake, but if I want to use Meson build system or SCons, Make and others I can do it as well. Look at the generators page: https://docs.conan.io/en/latest/reference/generators.html
I do not think fragmentation in build systems is even a real problem anymore because the "not fragmented but simple" philosophy leaves many things out of the box, such as compiling something custom when you really need it.
The system is very flexible. True that the learning curve is harder at first, but much simpler than it used to be in the past.
I can have full projects, not even with Conan, just with simple Meson wraps (Meson wraps are source-only dependencies though) pointing interdependencies to the same version of a library with a very reasonable amount of work: sometimes a single switch in the options, sometimes a small patch that you can keep in your subprojects/ directory or somewhere else.
I can easily generate .cmake and .pc files from Meson itself without trouble, which are two of the most used build systems. But if on top of that you build a Conan recipe then you can have many consumers of your packages for free.
It is nice to have a tool that is more ore less streamlined sometimes, but build systems is a difficult topic and probably there are some that do a few things better than others: cross-compilation, linux tweaks, generating VS solutions for you if your company uses VS, XCode or whatever. It is not bad to have a choice as long as you can interchange.
I would say it is even better than having the one-size-fits-all thing and later when you want to go to the real native stuff then you discover that your tool is too basic and cannot switch.