To whomever can take AND hold the armrest go the spoils of victory. What care I for plaintive cries of "fairness" when nature provided me the means to take the armrest I deserve from less capable arms?
I bet you get a lot of crotch in the face for your actions. Hot, seething, sweaty crotch from every person in the middle seat. The reason people choose civility is that eventually everyone benefits. Lack of civility immediately makes it worse for everyone. People in planes are already miserable, no reason to make it worse unless you enjoy causing others misery, which says more about you than anyone else.
That's why I always begin a flight by bludgeoning my neighbor into unconsciousness.
I personally smear my own feces on the armrests, having come prepared with plastic arm sleeves so that it doesn't get on me.