If a frame absorbs some of that vibration, it is naturally going to absorb some of your energy. So "harshest" ride quality isn't exactly the term I'd assign to aluminum.
Of course that would make wooden bikes a matter of taste, and not superior by any stretch of the imagination.
I pretty much only use fixed-gear bikes for city riding, even in cities like San Francisco and Seattle with tons of hills (albeit one with handbrakes — I'm not suicidal). I like the way they look and feel, and there are other benefits (generally cheaper, easier to maintain, etc) but the main thing I like is the "impracticality". Cycling is my main form of exercise, and on any given day the bulk of my moving-around time might be biking to/from the coffee shop or my coworking space. The inefficient energy consumption and increased cardio is a feature.
I think it's safe to assume each rider has a different utility function, in some cases drastically different. "Extraordinarily impractical" bikes may very well have other upsides for a particular rider that outweigh the disadvantages for them.
And some bikes are impractical in one way but extremely practical in another, like folding bikes. I have no experience with them myself, but I expect the riding experience to be mediocre. But the ability to fold them up and take them with you is fantastically useful.
Engineers have tried to measure efficiency differences between different types of typical road frames and been unable. This isn't really an important effect in terms of how fast you go, even if it feels better. Most of the energy in frame flex seems to be returned.