My daughter got really scared with the giant spiders in the forest and wanted to quit, but I managed to convince her to go on. She was happy we did because everything turned out fine in the end. She was bored by the battle at the end of the book, but we got through it.
It's hard to know how much of the story my son understood. I usually reminded him what happened last time every time we started ("They were walking through the forest, remember?" or "Last time, they just reached the mountain!"). The individual parts are easy enough to understand, I think.
They both loved the riddle game with Gollum, and it sparked (or fed) an interest in riddles in both. It's an excellent book when you want to take the step up from children's books.
I've seen 4 year-olds completely rapt while listening to a story like that and 8 year-olds unable to follow it.
When I looked into it, I found similar stories from educators of verbal skills not clicking until the 3rd or 4th grade, followed by a rapid catch-up and then zero lag. According to most special-ed folks I talked to, if there are more specific issues at a younger age (e.g. problems with reading comprehension, but not auditory, or vice-versa) it can be indicative of a learning disability.
You are 100% true about reading to them from a young age (and not worrying too much about "hard" words; kids don't know which words are supposed to be hard unless you tell them) makes a big difference. Seeing kindergarteners from homes without a single book in them vs ones read to from an early age is rather eye-opening.