Seems odd. Your doctor can't force you to take anything. If they say "do you want to try X?" just say "No". Not giving your doctor full medical context seems like a mistake - for example, maybe depression would be indicative of another issue, or maybe people who are depressed really shouldn't take a specific medication.
To each their own, and perhaps you have other reasons, but this seems like a less than ideal solution to a very trivial problem if the goal is just to not take an SSRI.