To you point about quantity. There are grains of sand. Does that mean that there are also numbers? I obviously argue no. You have to take some of the grains and set them aside to count them. You have to split your pile into two and recombine the two piles to add them. You have to line the grains up in order to number them.
By itself there are only grains of sand, mathematics only emerges when you impose additional structure on them and perform operations with them. There is no addition in a thousand grains of sand, it is the act of separating and combining them. Or at least imagining to do so.
You can build a huge part of mathematics on top of set theory but that does not mean that all of mathematics is already contained in every large enough set of things, at least not in any meaningful way. Only when you decide what to with the things in the set, what kind of manipulations you are willing to consider, that the structures mathematics describes emerge.
And while physics looks very mathematically, I see no problem with this. The most simple rules can give rise to staggering complex math. I find it much more reasonable to think - gross oversimplification ahead - that photons travel in straight lines in a three-dimensional space and therefore the electric field follows an inverse-square law instead of assuming that the universe already contained the mathematics needed to deal with electric fields.
There are just photons moving around, only after deciding how to measure distances and to look at spherical shells around charges we find a particularly simple way to describe electrical fields. But we could have made a lot of different choices leading to wildly different descriptions.