It's all self-justifying fantasy on Draper's part, and passive aggressiveness. He's a crazy old coot with dreadful personal hygiene who never brushed his teeth or took a bath, with an inflated sense of his self-importance to go with his disgusting habits, in addition to a predilection for "working out" with young men.
Years ago one friend of mine went to a conference Draper was at. During a break he went outside for a smoke, well away from the building. Draper left the building through the same door, saw him smoking, and did his "I'm going to scream at you now" act.
My friend had no idea who he was, and as a result almost punched Draper in the face.
Then there's his criminal record. But to Draper it's all a persecution complex. He doesn't have enough sensitivity as a human being to understand why many people didn't want to work with him at any age.