Versioning is a facet of documentation. SemVer is not the only way, but it's a widely-understood convection and quite logical.
The core issue generally comes down to: "If I upgrade, what do I have to do?"
If you have good release notes and upgrade guide(s) that don't assume every user is following each release closely, that's probably ok.
If your release notes are non-existent or just an export of your commit messages or ticket titles, SemVer helps cut through the noise.
Going from 1.2.34 to 1.2.85? No second thoughts. To 1.5.2? I'll probably look at what was new in 1.3.0, 1.4.0 and 1.5.0 just to keep updated. To 3.2.0? I'm going to be super cautious and assume I have a bunch of changes to make, and hopefully there's a 1.x to 3.x guide, or at least 1.x to 2.x and 2.x to 3.x.
In the worst case, it compiles fine but a bunch of stuff breaks at runtime. With nothing but SemVer, I'm at least prepared to test extensively. If I go from 1.2.34 to 1.2.x and get a bunch of runtime breaks, my first reaction is going to involve profanity and be something along the lines of "time to replace this garbage library with something better."
As library authors, we're under no obligation to use SemVer, produce release notes, documentation, or do anything else. The nice thing about SemVer is it's basically zero-effort yet provides massive benefits to users, especially if everything else I mentioned is lacking.