Let's say I release some code that is isn't fit for purpose. For example, it corrupts user data 5% of the time.
The code has a README that doesn't mention the issue, and I close all issues that refer to the problem. I also reject any PRs that try to fix it, and so on. I don't, however, disable issues or say the code is unmaintained, so there is a steady stream of people whose data gets corrupted when they download and use my code.
Since you don't seem to like the term "social contract", would you at least admit that I am violating a few norms and acting like an asshole?
The freedom to fork (for any reason) is orthogonal to the question of whether the maintainer is being an asshole or not.
Of course, people are also free to be assholes (for any reason). But it shouldn't be controversial to say "don't be an asshole".