If you're working for a big company with a monorepo where everyone is trying to live at head, maybe you have to care.
But if I'm giving away my software for free to anyone who cares to use it, I'm entirely comfortable in saying that if you relied on undocumented internals you get to keep both pieces when it breaks.
[1] which may hint at deficiencies in your library
Even this can be a breaking change for you. Suppose you're writing a bootloader that must fit in 512 bytes and you import a library. If the size of the library plus your code exceeds 512 B, then your program doesn't fit the target and will not compile. Hyrum's law is real.
Also remember it's not the size of the dependency that matters it's the size of the portions of the dependency used less any cross-crate optimizations. So you're always going to have to be careful and test regularly.
> since the compiler will already trivially catch such mismatches
That's too late — that breaks someone else's build. The goal is to check types before publishing an update.
People seem to give too much weight to versioning, but it's just a means to communication. Like any other communication mechanism, it's subject to error which should be somehow detected and/or corrected. And versioning only concerns with detection---it doesn't say what you should do when a breaking change is dropped! Focusing to the error correction is more sustainable and user-friendly.
The point is to catch changes that are part of the library's public API. That's something the author can be reasonably expected to evolve in a controlled fashion.
If you care about the internals, the onus in on you.