Yes and no. I was in freelance photography before switching to programming (weird, huh?). I thought repeatedly: I can't just give up photography, that'd make me a pussy. I should aspire to do what I love.

Then I dumped professional photography and became a programmer, and I am freer, better paid and happier than I would be otherwise. Quitting one thing let me find another that is better.

I guess, the real difference is that someone (Mike Rowe) tells you want to do: don't quit, it's better. But if you don't know what's better, then you don't know if you are quitting the wrong path to pursue a right path, or what. Decisions you make in absence of a judge are always uncertain. And who has the authority to judge me?