I was going to write something fairly audacious and start dragging in the history of Western philosophy, but instead I'll just focus on what I see as the flawed foundation of your argument: fairness.

I'll argue that fairness is not really an intrinsic understanding we all share but instead is a concensus. You could argue that it's not fair that some people are blind, but that you'd find few sighted people willing to put out their eyes. Fairness is a relatively fluid concept and it doesn't make sense to try and anticipate how "fair" your actions are.

The way out of this dilemma is avoiding following this line of thought all the way to first-world guilt. So in your case, you have skills and talents you know you are applying and skills an talents you know are idle. You also likely have talents yet untapped. It is a waste of your energy to worry about this.

Do the best thing you can that intersects with the things you are most interested in. Group concensus in the forms of laws, ethics, an morals should be consulted but not necessarily blindly followed.

Do good, and do it well.