I don't quite understand how you do not recognize that that is a strong argument against your position.
Rather than providing a basis of not allowing software to patented, I think it makes the statement "X can be encoded as math" trivially true - and therefore uninteresting.
In so far as there is value in patents on any systematic procedure at all, the position that "software should not be patentable because it can be encoded as math" is completely untenable.
Think of the guy who invented the equi-tempered scale. All throughout history you had to make instruments that are tuned to play in one scale. Now you can suddenly make instruments that can play all of them. Solves a real problem from music by making an improvement in the representation, so it can be viewed as all math. But is it really "just" math?