If the library follows the rules, then...
Separation of minor and patch versions doesn't make sense. Both in the case of a minor or patch version change the client should be able to upgrade to the latest version.
A major version change means that backwards compatibility is likely broken. In that case, we're actually talking about a new library.
Therefore you can just use a single integer as the version number, and if you break backwards compatibility, then you change the library name. If you had a library called MyLib with version 1.0.0, then you can just go with version 1 instead. In case of a patch (no functional change), you go to version 2, and all clients can immediately upgrade. Then if you add a new feature, but backwards compatibility is preserved, you can go with version 3, and all clients can immediately upgrade. If you break backwards compatibility, then you rename your library to MyLib2, so it's clear to everyone that this is a new library (with similar functionality).
The real reason for minor and patch versions is that libraries actually don't follow semver, and patch version change means 'small change, don't worry about re-testing', minor version change means 'not that small change, you should re-test' and major version change means 'we've very likely broken most dependencies'.
In real life, you need to re-test anyway.