A lot of the concern is essentially about the precedent it sets. If you start questioning the logic of criminalizing cannabis, what's next on the list? That's the fear driving a lot of the reactionary opposition to a lot of things these days, and I think especially in the tech industry we have to remember that a lot of people don't have our appetite for risk.
Personally I think we as a society have demonstrated a considerable capacity for adaptation, so there's little to worry about, but my extended family certainly disagrees. There are a lot of other linked issues that operate similarly and it's a bit of a tragedy.