First and foremost because you take them at a point in your life where you're, well, depressed (used as a blanket term here). So patiently waiting for the effect to manifest is a burden in itself, in addition to hoping it works and not being sure if you're experiencing a placebo or actual effect.
But what scares me more, personally, is that it takes such a long time to get on and ween off. It seems to fundamentally change something in the human body or brain. As opposed to e.g. amphetamines, and benzos, which may have long lasting effects from prolonged use, but otherwise appear to be pretty straight forward.
Having never needed antidepressants, I'm curious if others who have taken them, worried about this before or afterwards?