Personally I've found over the years that this balance between reading a very small number of books closely and a whole lot casually has worked well and has led each set to complement the other.
I'd not worry about "retaining" in the sense of being able to mouth all sorts of facts and figures and arguments and story details several weeks later. Sometimes this stuff is important, but for me anyway I only "retain" in this sense when I take the further step of taking notes after finishing a chapter, etc. Most books aren't worth that IMO given you can just pull them off the shelf when you need to look something up. But books can affect you in other dimensions, especially in slowly enlarging your perspective in ways you hardly even notice.
(But I mean, if you literally couldn't talk -at all- for a few minutes about a book you read last month, that's a different story!)