"It is in action alone that you have a claim, never at any time to the fruits of such action. Never let the fruits of action be your motive; never let your attachment be to inaction." Bhagavad Gita 2:47
"Besides, considering your duty as a warrior, you should not waver. Indeed, for a warrior, there is nothing but a battle fought according to dharma." - 2.31
Further, the reason for action is held up later. This is also the quote that Oppenheimer used to describe setting off the Atomic Bomb.
"Time I am, the mighty destroyer of worlds, and I come to vanquish all living beings. Even without your participation, all the warriors on the opposite side of the battlefield will be killed.
Therefore, arise and achieve glory! Conquer your foes and enjoy a prosperous empire! O best amongst archers, all your enemies have already been killed by Me – you are but an instrument.
Drona. Bhisma, Jayadratha, Karna and other heroic soldiers have already been killed by Me. Fear not – fight! You will certainly conquer your enemy in this battle." - 11.32-11.35
Not to simplify the complexity of the Gita but what I take from it is: We can only take action and in the long run everyone is dead so why not take righteous action along the lines of what you are good at and thus fulfill your duty to society and God.
Furthermore, the rigidity of the caste system is a modern invention. It occurred during British Rule (https://press.princeton.edu/titles/7191.html).