But at that point what you're doing has no resemblance to semver. And there is nothing at all wrong with that!
Just don't call it semver, because if you do, people will expect it to be... y'know... semver.
The same applies to putting your not-semver version number into places where people or tools expect a semver number to be placed. If you encounter those places (like npm) please put an actually-semver number in them.