I see a few reasons for this:

First, the books that introduce us to the world of literature tend to stay with us, especially those we choose on our own. This is opposed to the books we are forced to read because they are "great." For example, I still hate "The Great Gatsby" even if it is objectively great because I was forced to read it in a class with a teacher whom I did not respect.

So, I'm going to say Piers Anthony was more memorable to me than Hemingway. I'm sure that Harry Potter would have stayed with me if I'd read it at 12.

I watched through these lists and noticed that some of my best read friends still had YA books on their lists, so it doesn't seem to be "just" a matter of people who don't read much.