I do think that many posts in this thread are overestimating how hard it can be to get guests on podcasts. Some connections may have helped during his early episodes, but many of the guests on his shows (just scrolling through the episode list) were probably quite easy to book given the audience that they would be speaking to. I see a lot of people who would be actively trying to talk about their subject. Groups like academics and tech researchers are heavily represented in early episodes. Then later you start getting people who might be a "challenge" to book, but only after there are years of a backlog of interviews.
He then also has several interviews with people with their own large podcast/youtube followings which are always a large boost. Just scrolling through his video history I see people like Joe Rogan, Dan Carlin, and Jocko Willink as an example.
His interviews also hit many different interests, not too narrow of a focus, which makes it easy to build an audience based on somebody looking for any number of topics for a podcast/youtube video. That certainly helps build listener numbers over the 4 years that the podcast has been ongoing.