Isn't it fundamentally just about the ability to manipulate one's environment, though?

Such capacity is the only thing I ever envy about others: the power to craft their environment and sculpt their own experience to fit their desires, regardless of content. The relative difference is entirely incidental, since I primarily notice only others who are further up the hierarchy.

Is it just me? It seems like a major fallacy to think that in general people desire power over other humans as an end in itself.

Perhaps it's the technocrat talking, but the ideal situation seems to be Unenvied Power: pure power over one's environment, with zero conscious representation of that power in others' minds.