Challenging music is a tough sell because there has been about a hundred years of a critical mass of composers who took the metric of "challenging" and made it a target.

That created a lot of noise that's not of the compelling type. And while it's not so pretentious that, say, the music accidentally sounds outside of the human frequency range, it's pretentious enough to make non-musickers question whether there's really any there there.

Music cognition is still a young enough science to leave questions lingering about what is and isn't happening on a musical-perception level.

Also-- not sure I've ever seen this talked about, but music perception shares sensory apparatus with one's general sense of hearing. So challenging music with unfamiliar timbres and frequency combinations can be physically grating and overload the listener on that sensory level. It's difficult to be open to new, unfamiliar musical forms, and probably impossibly so if one is grumpy because the delivery format makes one feel as if the music is actively hurting one's ears.