I really enjoy reading fast books which have no value other than to make me chuckle or give me a thrill[1] where my aim is _not_ to be productive because well you know life isn't just a big ol hamster wheel ...
though I agree that most books which I read which stay with me mentally and where I see myself going back to them are far from easy. These are in the class of being at least x years old (Jacques Ellul, Marx, Mumford, Nietzsche, Schopenhauer, Kant, Zapffe, Heidegger, Foucault, Bertrand Russel, Cioran, Dostoyevski, Kafka, Orwell, Sinclair, J. Austin ...) and preferably in the public domain.
A book which I never read at least twice isn't something I'd consider having an impact on me. But then sometimes drifting aimlessly with "wtf did I just read?" moments has more impact/effect on me than anything that is listed as a great classic. So there is no reason to be elitist imho (and I can't overstate the _humble_ here since 20 years ago I thought Dean Koontz novels were the bees knees).
[1] "Bill Bryson", John Cleese, or the absolutely hilarious "Let's Pretend This Never Happened" by Jenny Lawson.