Agreed.
Its tough though... I know of a couple times in my life I've thought something was OK until I saw something else.
Examples, Glass. Binoculars, camera lenses, rifle scopes, telescopes. Glass differences are very real. You might be A-OK with some Bushnell Binos, but USE Swarovski or Vortex Razor or something lower-highend and you'll get it. Now, I didn't write LOOK THROUGH, I wrote USE. This has an obvious point of diminishing returns. I would never buy higher-highend glass.
Another example is something like TV black levels. If you don't notice black levels - good for you! Don't look for it. Because once you have a dark room and notice it, you'll see it forever. I notice soap opera effect, black levels, local/regional dimming, burn in, and it all removes me from the material. Again, Sucks. I wouldn't buy a higher-highend TV, but I pretty much can't get by with low end anymore.
So... I KIND OF WANT to believe there is a serious quality range with audio gear, but I guess the difference is in glass or TV black levels, I can snap a picture and prove a difference. Audio is only ever perception. I can't prove a wave on an oscilloscope is more pleasing than another exactly within reason.
That said.... You're right. Sterophiles and Audiophiles seem to be people with too many dollars and not enough cents/sense, and lots of people taking advantage of that. But maybe I just haven't used the right gear yet.