Yes, it does.
> Both in the case of a minor or patch version change the client should be able to upgrade to the latest version.
Yes, but if it's a minor version change then, as a developer, I also want to review the changelog for new features I should consider whether we should add backlog items for refactoring to take advantage of, while a patch doesn't create that need. There is more space for SemVer-aware tooling than “is this safe to update”, though that's obviously the most important single piece of information.
> The real reason for minor and patch versions is that libraries actually don't follow semver, and patch version change means 'small change, don't worry about re-testing'
No, you always worry about retesting because SemVer is a statement of API intent, but does not include a guarantee against bugs, including new or modified bugs.