This is quite the straw man argument. Most of those people either support this already or just have no opinion on it
Source: https://www.nytimes.com/2019/03/25/nyregion/new-jersey-marij...
Abortion is a pressing practical issue People may be very vocal about it.
Psychedelics is mostly an issue of recreation, even without a few millennia of consumption ingrained into culture like alcohol.
So the efforts people exert to legalize each are commensurate to how pressing each issue is.