Even if legal prostitution led to higher crime, more violence, death and disease (none of which is true, of course), the issue is still fundamentally one of principle. Why does the state have a role to play in a transaction between two consenting adults? The only sensible argument would be that the state has a role only insofar as consent is in question, and therefore maybe assists in some capacity in terms of ensuring consent. Beyond that, we simply should not be legislating in search of outcomes.
In much the same way that alcohol prohibition might lead to lower rates of violence, rape, obesity, cardiovascular disease, unemployment, unwanted pregnancy, etc...we oppose prohibition from a principled position. Just as you might argue that putting a security camera in every room in every house would vastly remove unresolved criminality, false imprisonment, domestic violence, etc...we oppose such an invasion of privacy from a principled position.
It really doesn't matter how sex work is actually affected by the legality of it, the state simply has no right to interfere in a consensual transaction which does not produce negative third party externalities, period.