Just do what you love. Period. I've had times in my life when that was programming - and it got me a long way, and I loved doing it while I did it. Then I felt the challenge of learning how to build a business. So I made a choice, and stepped away from coding, and started pursuing my passion of building businesses. There's nothing wrong with your passions changing - think of it as opportunities to learn new skills.
One of the funny (paradoxical?) results of this is that I've had my eyes opened on things that I'd love to code. So, I've gotten passionate about programming again. Ain't life funny?
You don't need to apologize for being honest and self-examining - you only need to apologize if you aren't following your love and passion.