Yes this is exactly right: "The physical intepretation of a negative number is the quantified absence of a unit." In the physical world, the absence of a unit just means the perspective doesn't include whatever it is that is intended to be observed, although that thing may exist elsewhere. In other words, a negative number may be an abstraction for quantifying that a unit has left an isolated system, but I'm not sure our current usage of numbers as abstractions captures this - it seems like we often use negative numbers (in any unit) to denote the actual subtraction of that unit universally as opposed to its transfer to another isolated system. (This could be where mathematicians can get into trouble compared to physicists as perhaps they don't have the same habit of observing units.)