You could say the same thing about a person's right to their own body. If you don't consent to their body being theirs, there is no rape or violence when you touch them without their consent. Bodily autonomy is an illusion according to this malevolent, bad faith argument.
It's self-evident, from the perspective of any coherent moral framework, that people have a moral right to exclusive access to their own body, and the wealth they produce or acquire in voluntary trade, and that the people who disagree with this have no respect for others, or regard for the best interests of society at large.
Just think how absurd it would be in the Fable of the Ant and the Grasshopper, if the Grasshopper used your argument to demand the Ant forfeit the food he had painstakingly stored up for the winter, and the disastrous social consequences if this pathological moral framework were socially accepted.