Don't judge all of us audiophiles with a blanket statement. Yes, measured hearing capacity is definitely a key metric to know (why invest in super tweeters when you can't hear past 18KHz for example), but room acoustics is probably the very next thing I'd look at.
I've seen so much "expensive" gear in rooms that are absolutely horrid in terms of sound reflections. Just a carpet, curtains or a couple of cheap sound panels can make wonders in improving sound if you know just a bit of the physics of sound. You don't have to invest in super expensive equipment to do that.
Furthermore, there is a healthy DIY community these days, where you can build your own speakers, cables, amplifiers, etc, with very little effort, and get a lot of enjoyment in the process and with the product. I built a set of LX521.4 and LX Mini speakers from Siegfried Linkwitz [0] and it changed completely the way I listen to audio now for an order of magnitude less money that what I'd had to pay otherwise...
[0] http://linkwitzlab.com/