Why does it bother you so much when people say you have privilege? Your privilege is that your passion and hobby is in high demand... most people don't have that. They have hobbies like fishing or golf or gardening, and those things aren't in high demand. If you had been born 200 years ago, your passion might also have been something that wasn't in high demand.
I don't see how that concept upsets you. No one is saying you did anything wrong, or that you haven't worked hard. We are just asking you need to accept that some of your success is due to luck, and your luck is a privilege.
Why is this important? Well, it is important to have an appreciation that not everyone had the same luck, and other people not doing as well in their careers is not necessarily because they haven't worked as hard as you.
Once you accept that, the next logical step is to realize that those of us who have been lucky (like me and you, who were lucky enough to have our skills be in high demand) should think about ways we can help those who haven't been so lucky.
A healthy society takes some from the lucky ones to help the unlucky ones. This doesn't mean that everyone should get the same reward, or that you don't deserve success or praise for your skills, just that there is some humbleness in not thinking you are better than someone else who is not as successful.
The fact that you get so angry when anyone points out that some of your success is based on chance and was not just because you worked harder is troubling. Why are you so defensive? Is it because you want to be able to think all of your success is based on your own hard work and therefore you don't need to share any of your success with the less fortunate?