The problem is the notion of victimless.
There are crimes that put people at risk but don't always create a victim. Do we allow for crimes that have a risk of a victim even if there isn't a direct victim?
There are crimes where we say that someone is a victim despite their own feelings of being victimized or having suffered a crime. Do those stay as well?
We have crimes where there isn't someone physically hurt and not a direct theft, but there is a reduction in value of items. Do those crimes stay?
Take zoning laws. Is violating a zoning law victimless? If I open a foul smelling poultry factory in the middle of a downtown commercial zone, it could greatly negatively impact that commercial zone to the point where businesses would not be able to afford to stay open. But if that makes it a victim crime, then what about when homeless people living in that same area also drive away customers to the point it no longer functions as a commercial zone?
As for drugs, what happens when companies start selling direct to users? Antibiotics and opioids without needing a doctor's prescription. We already have seen the problems even when doctor prescriptions are required, imagine what happens if drugs are declared a victimless crime and no longer prosecuted.
I think the notion that some crimes are victimless is based on looking at specific incidents that don't have victims and not considering it at scale. Which is to say that the example by the other poster of drunk driving might be a much more fairer comparison than what people originally took it as, because while the relation between the risk and possible victim is easier to understand, the question is if such a relationship is consistently used to justify a victim, not if the relationship is easy to understand.