His prior fame undoubtedly helped him get off the ground, but if you think the show's success coasts purely on his earlier fame, I'm afraid to say this is a story you're telling yourself, seemingly in part, to make yourself feel better.
A podcast doesn't stay relevant with a large and loyal audience unless it's actually good. To pretend like JRE is a continued success purely because he was on TV before 1) ignores the countless podcasts started by people far less famous than Rogan was when he started which have failed to take off and 2) sounds like a cynical, resentful thing you tell yourself feel better along the lines of "well, if I came from where Rogan did, I'd be just as successful."
Whatever you want to say about the reasons anyone tuned in to begin with, it's transparently unfair (and betrays your own "issues") to not offer a balanced perspective which includes the need to acknowledge the obvious which is that Rogan's success just might be _also_ due to the fact that he is a good interviewer, that he has good guests on his show, that he's able to spark interesting conversations, etc.
The idea that his TV stint and MMA fame makes a listenership of 12M an automatic thing is just laughable.