Clive James led off an essay by suggesting that there was a buddy movie to be made around Wittgenstein and Russell. He was joking, but part of the joke is that their fame much outran any engagement with or even interest in their philosophical work. I don't think that leaving little memory beyond the work is a bad thing. What does it add to one's understanding of Kant to know a few anecdotes about the regularity of his life.

One can regret the opinions in Frege's diaries without it affecting what one thinks about the work.