Those that read 'The Book of the New Sun' will know the feeling
I feel guilty mainly when I run into someone else who says they love the book, and I am totally unable to have a meaningful conversation about it because to be honest I didn't understand or retain much from it. And I end up looking like a poser a lot of the time I'm sure, and maybe I am in some way. But I still read and enjoyed it!
As for looking like a poser - most people will respond well to "tbh I didn't really understand what I read, tell me more and I'll keep it in mind if I revisit" (or "help me understand this other thing I have some concrete memory about", etc). Some jerks will scoff or dismiss you, most people I've encountered are pretty open to a good discussion even after it's been revealed.