We don't see people arguing we shouldn't patent physical machines because ultimately each operation is a simple physical movement, so the sum of everything is just a sequence of physical movements.
Yet we see people arguing we shouldn't patent software "machines" because ultimately each operation is a simple mathematical operation, so the sum of everything is just a sequence of mathematical operations.
The argument against the patentability of maths, AFAIK, has always been "maths is naturally inherent and merely discovered by humans". This, to me, is an argument against pure maths especially; but as we get into applied maths I think it's much less arguable that algorithms and data structures are naturally inherent (much like how physical constructions—machines—aren't naturally inherent from the laws of physics).
I still strongly think far too many patents get issued, but I'm less convinced than I used to be software patents are inherently bad, and that patents do need reform in various ways.
Then you are against software patents.
If millions of people carried machine shops around in their backpacks, you would be against mechanical engineering patents, because far too many patents would be issued.
At the core of patent law is "presumption of validity", which places the burden of proof on the accused, rather than the accuser. Anything that multiplies the number of patents the PTO has to examine makes that presumption less reasonable. At some point it's arguable that a party accused of patent infringement is not getting due process.
Software patents. Presumption of validity. Due process. Choose any two.
Also, IMO, a formula, which tells you what to compute, is essentially different from an algorithm, which tells you how. FFT would be patentable because it is non-obvious, even when given the formula for doing a Fourier transform.
It still might be non-patentable on other grounds, such as the fact that Gauss apparently described it in 1805 (http://www.cis.rit.edu/class/simg716/Gauss_History_FFT.pdf), two years before Fourier published his work on what now is known as Fourier series.
A mathematical formula is just a symbolic expression of an abstract concept, with the procedure (algorithm) how to use it being implicit (= obvious to the person skilled in the art).
A computer algorithm can be trivially converted into purely mathematical/logic representation ("formulas" if you want), e.g. using things like lambda calculus. And vice-versa - an abstract mathematical problem formulation can be converted into an algorithm (assuming the solution is known).
If you start patenting algorithms, you are patenting math.
For example, you could write a formula to express the property of order in lists, and maybe with some mechanical procedure (falling under "obvious to the person skilled in the art") you could then generate an algorithm that sorted lists. However, what's the time/space complexity of that algorithm? What are the practical runtime characteristics? For industry applications the practical runtime characteristics matter as much as anything else. That you technically could compute something doesn't matter at all, if that computation might not finish until relevant celestial bodies have phase changes.
I'm not saying Quicksort should be patentable, but I do think it's at least a step and a half removed from pure math.
When the people who spread adoption ignore the law, sure patents don't prevent anything, e.g. https://en.wikipedia.org/wiki/MP3#Internet_distribution
If you take a different example, say, improvements to JPEG, it's a lot harder for camera manufacturers to ignore patents. You're at the mercy of the patent holders, whose terms apparently haven't been reasonable enough for anybody to improve the compression in digital photos beyond what we had decades ago.
Whether it's a good thing that patents exist is another matter, though.
Magsafe connector is an engineering invention that doesn't exist or derive from anything in the nature.