If you have four oranges, the quantity "four" is right there. If you take away one of those oranges you know that the result cannot be split evenly without a remaining orange because of the properties of odd numbers.
If you cut the remaining orange in half then you get a rational number, but is that self-evidently real? The halves of the orange are only "halves" because we consider them in relation to their origin, which we consider to be "one" orange. So rational numbers necessarily involve the human action of relating some quantity to a reference quantity, therefore they are a higher-level abstraction built on top of the fundamental physical property of quantity.
In the end I decided that math is based on a foundation of quantity (and maybe "space" as well?) and everything else was a derived abstraction. I am very curious if anyone else has a good argument for other parts of math being fundamental.