Some people believe sex and gender are different things and gender is just sex based stereotypes. These views are legal and in the UK considered protected.

You wouldn't call someone transgender if they didn't believe they were trans, or someone a christian if they didn't believe in the bible, it's the same the other way round. If someone doesn't think gender is real, calling them it is putting a label on them they don't agree with.

I've then seen the label used to devalue people's opinions on things, so yes it can be a slur. Unfortunately these debates end up being so toxic due to people on both sides.