It's even better than that. Working from the other side, we generally take the natural numbers N = {0, 1, 2, 3, ...} to be "real" in a philosophical sense as we have an implied mapping of x in N fundamentally describing possession of discrete quantities of goods. This get a little more tricky when you generalize it to the integers Z = {..., -2, -1, 0, 1, 2, ...}, but we still consider these physically grounded as you can "owe" a discrete quantity of goods to someone. This is an incomplete mapping to reality as possession of y goods where y is in Z- doesn't actually describe where your y goods go to, but it's useful enough that we ignore that.
Now that's all good and well for discrete quantities of goods, but what about fractional quantities? We need a new system to describe more numbers between the numbers we already have. Thus we defined the set of rational numbers Q = {n | n = p/q where p, q are in Z and q is not 0}. This lets us compactly describe almost any number we want between the existing integers. Most still consider these to be physically grounded, because we created these to describe concrete things in our reality which they do very compactly. You can claim that at least some of these are physically grounded in reality as using 1/3 a cup of flour or buying 1/2 a watermelon is certainly something one can very clearly and explicitly do.
The rationals have some issues though, namely that they still have holes in them. Suppose you want to describe the relationship between the diameter of a circle and its circumference. The constant you use to transform one to the other, π, does not exist in Q. You can get as close as you like, but you can't actually reach it. That's a bit of a problem for people whose job it is to make sure that the things math says are correct are actually fully correct. There are other problems, suppose you want to make a rectangular plot of land whose area is 2 square miles. How long does each side need to be? You can get as close as you like by using rational numbers, but the actual length of that side (sqrt(2)) is not in Q.
To fix this, we then very delicately construct the set of real numbers R = {x | where x is in {Q and all of the numbers described above which are missing from Q}}. This is where the physical grounding of the numbers starts to get ugly, because as it turns out that just like for some numbers in Q (consider 22/7, which does not evaluate to a fixed number of digits but rather goes on forever) these are uncountable and most of them unrepresentable, i.e. if you tried to write the number out completely on a piece of paper the universe isn't big enough to hold it (and in some cases, you can't even specifically refer to the number in constructive terms as with π). This turns into a whole philosophical debate which IMO is silly but some people do take pretty seriously.
But wait, there's more! The field of real numbers is closed under addition and multiplication (and thus subtraction and division), but it's _not_ closed under some other operations. Suppose you're an EE trying to represent physical signals that very much do exist in reality, and you need to take the root of a negative real number because that is a meaningful quantity in context of the still physically grounded thing you're modeling. Well you can take the root of a negative real number, but that number is not itself a real number. Thus we must define the imaginary numbers to hold those negative roots and then the complex numbers to join the imaginary numbers back to the reals.
In every single one of these steps, the new numbers were created to describe aspects of our observed physical reality. The break from the common definition of "numbers are real because I can go buy 24 tangerines and 24 is a number" happened way back at the integers where we added a reflection around the end of the natural numbers. From a perfectly reasonable perspective then, the real and imaginary numbers are both "real" in that they describe actual physical things that exist even though you can not in fact buy -1 apples or 22/7 cats or π bananas or sqrt(-1) movie tickets.