There's a pastor who sometimes answers this question with "I'm a doctor for souls." .... The answer "Pastor" can be a conversation killer.
I would have to hold back to discuss my views against religion with a pastor.
I imagine 'doctor for souls' is intended to show humor and approachability, but as you say, it would not work that way for me.
As it maybe should be. I'm certainly not denying that people identifying as pastors can offer solace and comfort to people in times of trouble, but they are doing so in the context of mysticism, in a particular form that has been catastrophic for many groups in our society.