Just to make sure I follow, do you mean the "music is math" idea fails, or that my question does? I'm not saying math has nothing to do with music. Frequency ratios and rhythmic divisions are clearly there. My point is that you can have deep musical ability without ever thinking in those terms, which suggests music is more than the math that describes it.
Sheet music is no more music than mathematical notation is mathematics. (Indeed both notations have roots in the same kind of two-dimensional seventeenth century calligraphic tradition.) However, as you point out, mathematics is indeed the study of patterns, structures, and relationships, and music is not. It is a structure and has relationships, but it is not a “metaphor of metaphors” as mathematics is. Case in point: modern visual art post-photography is much more affine to music than mathematics is, because after depiction became not quite trivial but literally photographic from the daguerreotype onwards, the role of the visual arts became the solicitation of emotion and the conveyance of narrative. Music has syntax, indeed it has very many, but it is not the study of syntaxes in the manner in which mathematics is.
I’m saying that though argument by analogy is very persuasive, “math is music” is a metaphor that is fundamentally unsound as the grounds for reasoning from. I’m not talking about the mathematical aspects of music (which, as you correctly point out, can be found to exist) but that music is performative and aesthetic, whereas mathematics is all about internal consistency.