Sorry what? Ancient humans invented symbols to count. How is that not symbolic?
Geometry is “attached” to the physical world… but in an abstract way… but you can point to the thing your measuring maybe so it doesn’t count…
Abstraction was perfected if not invented by mathematics.