I am a really mediocre player. But one can be a mediocre cook and understand and appreciate fine cuisine, or a couch athlete and understand the subtleties of football strategies. It's the difference between passive analysis and action.
And I certainly understand if someone dislike jazz. But saying it's not music is like saying Picasso's paintings are like a three-year old's drawings.