When I read that book I had to take some of his statements with a grain of salt. Even then, when the Posen proof was not yet known. As explained in this article he lied a lot. He had to, esp. the Posen incident where he got away with it even if it was totally unbelievable. We only got the proof recently. And his involvement in Todt's slave labour "Organization". He was his boss after all and got all the credit for its success. Even rising up to be the designated follower of Hitler. Todt got the death sentence for it and he successfully talked his way out of it.
But then his made up prison stories to impress his fellow germans just broke it. He was a lousy architect after all, as Poelzig correctly found out. What was interesting for the americans then, how he managed to be so incredibly successful as manager of war-time economy. It was unbelievable and unexpected. McNamara would have been very proud of Speer's accomplishments. He was a kind of genius figure for the americans.