This is it. Inverse is a property of functions or other relational operators, not "static" individual objects. You need a direction in order to invert it.
If we're talking about trig functions, inverse sine or arcsine is very different than just the reciprocal of the sine.
If we're talking about images, inverse usually means "rotate 180 degrees" or "color inversion".
Similarly, the additive inverse of a number m is the inverse of the function that adds m to its input.