> There's probably even a name for it as a logical fallacy...

Maybe: https://en.wikipedia.org/wiki/Argument_to_moderation