Sheet music is no more music than mathematical notation is mathematics. (Indeed both notations have roots in the same kind of two-dimensional seventeenth century calligraphic tradition.) However, as you point out, mathematics is indeed the study of patterns, structures, and relationships, and music is not. It is a structure and has relationships, but it is not a “metaphor of metaphors” as mathematics is. Case in point: modern visual art post-photography is much more affine to music than mathematics is, because after depiction became not quite trivial but literally photographic from the daguerreotype onwards, the role of the visual arts became the solicitation of emotion and the conveyance of narrative. Music has syntax, indeed it has very many, but it is not the study of syntaxes in the manner in which mathematics is.