I vaguely recall an anecdote being told here on HN where Donald Knuth met Steve Jobs. Steve Jobs said something along the lines of "I admire you very much, I have read all of your books!", to which Donald Knuth replied something to the effect of "I don't think so." (more politely, though, I presume).

Which is to say I don't think many people have actually worked their way all through them. Neither have I, there's no shame in that.

And, as you point out, it's not too late to give it a try.