There are lots of problems with single-file libraries. Here are a few but I'm sure there are others I've forgotten or haven't thought of.
* You have to wait for compilation of the whole library at least once every time you do a build of your program - certainly every time you do a clean build, and potentially even incremental builds if it's header only.
* If the library is header only (many of the linked libraries are) then you you potentially have to pay that compilation cost more than once per compilation of your program - once per every one of your source files that include it.
* Again this is specific to header-only libraries, but to avoid code bloat you'll need to turn on link-time optimisation which is far slower than just allowing the linker to do its job by only compiling definitions into a single object file. (Admittedly LTO is a good idea anyway, but adding a bunch of duplicated symbols is avoidable extra work for it.)
* Some useful libraries are realistically just too big for their authors to write the whole thing in one file (e.g. protobuf, opencv, ... in fact most libraries I use on a regular basis seem to fall into that). They could "release" the library in single-file format, similar to SQLite's amalgam, but then if there are any problems (either a bug in their code or something in your code that makes you want to look at their code) you're now not looking at the original source but some mangled version of it.
* If the library is so large that its interface needs to be split over multiple headers (think Boost or OpenCV) then you're now bang out of luck. Hopefully the library has cleanly-enough separated modules you could potentially release these separately (e.g. OpenCV core, imgproc, imgcodecs, highgui, ...) but then you're essentially back to multi-file libraries.
* Adding a library with a lot of its own transitive depedencies takes proportionally the amount of effort as the number of those dependencies, rather than being handled automatically.
One interesting thing about all of these problems is that they get worse and worse as you need more libraries in your program, or need a larger library for your program. In contrast, using a package manager (I'm thinking particularly vcpkg here) tends to add a one-time cost at the start but allows you to scale your dependency list almost for free.
If you're writing your own library, rather than a application, then there are even more problems with this approach, but I won't quite open that can of worms.