When you buy it from dangerous people, you have to be in contact with dangerous people. It gives dangerous people leverage against you and people who care about you.
Dealers don't assault, endanger or attack random customers who just stop buying.
That's totally absurd, and that someone would suggest it - indicates that they are getting their information from Netflix/Hollywood films or something, not reality.
The vast majority of dealers are actually fairly normal people anyhow.
Doctors in emergency rooms usually have instructions to confiscate any illegal drugs they discover on OD-patients, but are faced with the very real dilemma of leaving the patient at their creditors mercy. This can cause real harm (as in "First do no harm") to the patient and their prospects of recovery or harm reduction.
But nobody is 'going after' someone who just decided to stop smoking something.
The counter of this is why underage drinking is higher in the US than elsewhere. Society made it a taboo and that attracts people. In countries with lower age limits there's less underage drinking. The culture itself doesn't see it as such a big deal and there's no pent up against.
Decriminalization means it's not illegal to possess, not that it's legal to sell. An important distinction in these sorts of discussions.
I would suggest that maybe it's ok to continue to criminalize the sale of the more dangerous stuff, say of Schedule I drugs, but... well, marijuana is still on the Sched I list, so it's not like that list is particular scientifically rigorous.
But it goes with greater awareness of the risks as well, and once it's in the public, you can expect a lot more discussion about its usage rather than a too simple "it's bad" attitude.