Abstract logical frameworks are not pulled from thin air. Human brains develop such frameworks with a physical brain that receives sensory input from a physical world, and hence there is a deep connection between the physical world and the abstract logical frameworks in many ways.
I remember struggling as an undergrad to come up with a metaphor for visualizing what a group homomorphism is, aiming to develop better intuition when working with them. It's hard because the real world doesn't contain any, and all the "it's kind of like X" types of examples you can think of aren't really useful when mapped back to the mathematical domain. If math described the real world, I'd expect this type of thing to be much easier.
(Though it is very interesting!)
But I'm sure pure mathematicians would also be totally fine studying randomly generated rulesets. The only restriction is that they should not lead to immediately obvious contradictions, since those would probably be kind of boring!
I do agree, but that we don't do that in reality is exactly my point.