Let me try to be more precise, I probably did a pretty bad job explaining what I really want to argue for.
I want to argue that whenever you think that there is some math already contained in the universe, it is actually you accidentally introducing it without noticing it. Look, trees! One, two, three, ... See? There are numbers in the universe. And look, I can add the number of trees in this wood and that wood.
It is of course tempting to conclude that the math was already there. But I argue that it was not, that it is you inventing it. It is your concept of object identity that defines what a tree is and when two trees are the same or different trees, you divide the world into different objects. It is your concept of grouping objects into sets. It is your concept of pairing the elements of two different sets to compare their cardinalities. It is your concept of including one more element in a set that defines what counting means. It is your concept of combining two sets that defines what addition is.
That are all things you can do with objects in the universe, but nothing of that is inherent. If you look carefully enough, all the things that look like math already exists in the universe actually require you making use of a lot of abstract man-made concepts. Many of them, like object identity and counting, are just so deeply ingrained in us that you don't notice that you are using them if you don't pay careful attention.
That is I asked for the empty universe, I wanted to take away all the objects on which you could accidentally impose structure. If math were truly fundamental, if Platonism were true, then seven should still exist in an empty universe. I don't see how this would be true in any meaningful way.
If we added a mathematician to our empty universe, he could invent math. He could come up with the idea of objects and identity and set of objects and he could think about operations with those objects and consequences of the definitions he made. But it would all have no relation to the otherwise empty universe, it would be a game of formal rules, mostly manipulating strings of symbols. If the mathematician had great imagination, maybe he could imagine actual things existing in the universe, how he could label them with those numbers he invented, and how he could visualize addition by moving things around.