It's not completely unreasonable point. I do see it's a very different type of work compared to mine. But I learned that life is not about doing something you love, but learning to love what you do. So I'm game to try something new.
I think you've nailed the point that they missed. You do not always get to do what you already love, especially when you have responsibilities, family etc. What you can do though is what you suggest, learn to love what you do. Which usually starts with being competent at it.