The reason for the whole prioritization of kindness generally is this: I have learned for myself that God is real, and our choices really do matter in this life and beyond, and the way I learned it means many other things follow as a result: the need to love God with our all, and our fellow humans as ourselves; and the importance of service to others, in the growth process we are all experiencing.
I also appreciate your point that you are only responsible for your own emotions. Sometimes remembering that helps me too -- especially if I consider those emotions to be a choice, or something that the person can choose, if they choose to act in that regard, rather than just being acted upon. Somehow balanced with the importance of serving/helping others appropriately when possible. There are a variety of situations of course, as you imply.
And just a quick aside as it occurs to me: I found from experience that if the only feedback is negative, the recipient will likely eventually quit listening or quit participating, or become destructive in some way. I think we all need a positive concept of future possibilities, i.e., hope, in order to move forward. Part of a leader's job is to be to motivate, and/or to know when that is needed, and providing that hope is important. There is a short book ~ "Leadership and the One Minute Manager" that talks about 4 different kinds (maturity levels) of reports, and the different treatment each needs, depending on their high or low levels of confidence and competence.