I think something that went unsaid here is that readers have fundamentally different reasons to pick up a book. Someone like Tyler Cowen, or indeed many commenters here, are quite keen to absorb knowledge, and time may be limited.
I enjoy reading for learning, but I also enjoy reading to quiet my mind, to practice focus, and to bring myself joy from the beauty of a tapestry of words. I agree with you that some of these objectives can’t be accomplished by ruthlessly discarding a book that doesn’t immediately captivate.