Ok, I was asking because if you meant "math" as in "geometry" you could just go directly into how OpenGL does his stuff or many of the higher level graphics languages like (the preivously mentioned) Processing or three.js.
BUT, if you mean "math" as in "abstract math" (which I believe is what you do) I don't think there's anything suitable for that ATM. You see, the thing is that computers behave in a certain way and languages/libraries are pretty much just tools that help you abstract away all the boilerplate associated to the many common tasks that you may want to do in a particular subject. That said, you can't easily escape the underlying paradigm, the "way of the machine"; which in this is case is pretty much "light on and off particular pixels on the screen in order to create shapes/effects".
There is not, however, anything stopping you from creating a new set of tools designed to do exactly whatever you'd imagine these tools should be. For what I see, you are at one of the most important stages anyone has throughout his life in order to innovate, that is, when you're barely starting and your mind is full of your ideas instead of someone else's. Take note of everything that comes up to your head and try to find a way to code it later, my only suggestions would be:
1) Don't give up, it's gonna be a very long and tedious process but don't give up.
2) Try to find the tools you feel are helping you in solving this problem, never the other way around (i.e. trying to adapt the problem to the tools). If you ever feel that you are not making any more progress with the toolset that you chose and have better options in mind just go for it; even if you have to recode everything from zero again just rinse and repeat until it's perfect.
You could be at the start of somethin very interesting, who knows; best of luck and don't forget to post your progress back here.