You are mixing up the mathematical universe with the physical (also known as real) universe.
For a mathematican 42 or -42 makes sense - there are no units necessary. The mathematical intepretation of a negative number is just being element of a set.
For a physicist 42 or -42 are pointless, because a unit is missing. The physical intepretation of a negative number is the quantified absence of a unit.
- by the way: https://xkcd.com/435/