Which philosophers are saying this? I majored in philosophy, still read philosophy (both ancient and modern academic), and have never encountered anyone saying that classification of things is a violent act.
It's true there has always been a struggle, going back to ancient Greece and Thucydides, about whether politics should be about justice (or morality) or power.[0] Between what is right, and who has force. And some, like Chomsky (who some might call a radical), say that everyone living in a modern society is complicit in some way[1]. But I've never seen anyone say that believing in justice/morality instead of power makes one complicit.
[0] https://plato.stanford.edu/entries/realism-intl-relations/#R...