I try to read a few "classics" now and then, and I definitely read more than just sci-fi. I like history, philosophy, biographies, etc., as well. But you touch on something we all have to struggle with: Time. There are only so many hours in the day to read, and every time I sit down to read, I have to make a decision on whether or not what I'm thinking about reading is a good use of my time.
So... I'm going to get a certain amount of raw "escapism", pop-fiction stuff, that I need just to stay sane. That's a given. Now, how much time do I have left to read Homer, or obscure French and Russian "literature"? And to what end? Just to impress hipster douche-bags and self-appointed "intellectuals"? Or because reading that stuff is actually going to add value to my life? Is reading Madame Bovary or Anna Karenina going to do more for me than reading Teach Yourself Haskell in 21 Days or Asimov's "Foundation Trilogy" books, or whatever?
The last few books I've read (other than technical books) include Run by Dean Karnazes, Born to Run by McDougal, The Belgian Hammer by Lee, Eat & Run by Scott Jurek, Racing Through the Dark by David Millar, Slaying the Badger by Moore, Running on Empty by Marshall Ulrich and The Secret Race by Tyler Hamilton. Would I be better off if I'd spent that time reading "literature" instead? I don't know, but I'm sceptical.
OTOH, I have a whole pile of stuff lying here by Nietzsche, Kieerkegard, Hume, Foucalt, etc., that I'm planning to work through. But I'm reading it just because I find it interesting, not because I see it as having any more inherent value than "low brow" literature.