In fact, Pale Fire is quite a straightforward story, with a creative form with some puzzles to decipher if you feel like it. Or you can brush past them; the suspicious presence of contradictions is a delight in itself. It was fun for me at 22 but at 33 I would not go back and read it—I don't recall it being particularly deep, but it was a good time. I'm more interested in the other Russians now. Not that Nabokov, Tolstoy, or Dostoevsky have anything to do with each other. One would never think to group them at all if they hadn't written in the same language.
Personal share ... I had the same hesitance about getting into Tolstoy. I read The Death Of Ivan Ilyich -- it's short and has few characters. Absolutely loved it. That sealed the deal so I'm working through Anna Karenina now. I'm reading the Rosamund Bartlett translation and it's just superb so I don't really care about the torrent of names and characters. It's worth it IMO.
also Pushkin's Queen of Spades
Think of how different the world would be if all Beethoven composed were Für Elise-type pieces, and he never composed anything as complex--and inaccessible--as his Grosse Fuge.
In my experience, my favorite books have been the ones that required me to work a little.
I think the problem is some people expect to not have to do any work when reading literary novels.
Your initial premise: "I find that literature that requires work does not provide more food or insight into the human condition..."
Your conclusion from this premise: "If complexity is not what makes the work valuable, what purpose does it serve?"
And you answer your own question with a cynical take: "I posit that for some people complexity for the sake of complexity _is_ the appeal."
This makes me want to kick myself for reading Hacker News to be illuminated about the arts.
Still, I love user lermontov!
The point you might be missing is that some people might enjoy hard things not only because they are hard, but because only through their laborious efforts is the true brilliance of the work revealed.
(Edit: re-reading what I wrote earlier I can see how it can be read that way. My bad. )
And for what it's worth, there's plenty of modern-style "hard" novels that I've enjoyed immensely as well, including House of Leaves, Infinite Jest, Shogun, The Pillars of the Earth, and [pick any Neal Stephenson novel published this millennium].
Nabokov's stories and reputation are intimidating, but if you drop your anxiety over missing the point, he's still a very entertaining, passionate writer. I definitely missed a lot when I read Pale Fire, but I still enjoyed it a lot, even being naive and taking everything at face value.