I think leaving your comfort zone forces you to grow, so even if you sustain some short-term bruising, the overall growth makes it worth it.
I knew I wanted a career in software, but I figured that out just as I finished a completely unrelated degree (business, as it happened). Eventually I decided I was just going to make it happen somehow, bought some books, and started jamming the knowledge into my head.
It was hard at first, but I really loved it once I got going. Way outside my comfort zone, and I felt a bit insecure posting to forums with my complete newb questions.
After really getting serious, with maybe eight months of coding every day after work building stuff I thought was really cool, I said all right, screw everything, this is for me. I did something terrifying: Gave my boss six months notice and started saving for my move to the west coast.
After I moved, I wanted a break from working for the man. I worked for myself another half a year building iPhone apps, sharpening my skills, and generally having a good time. About the time my savings ran out, a Bay Area startup hired me to do product stuff and now, voila, I'm in the industry.
Funny thing is, I miss writing code so I still do it at night after work.
A lot of time out of my comfort zone but a lot of growth, too. And if I hadn't forced myself to do it, I'd still be stuck in Florida, doing work that didn't really connect with what I care about.
So I dunno, I think if you get the sense you're out of your comfort zone, that's the feeling of your mind straining to grow to meet your new challenges. So far, it has worked well for me.