Not to say that $80 million/3.5 million listeners is anything to scoff at, but it's hard to argue that his cultural significance is vastly diminished. Others have said he was reaching
25 million listeners at his peak so 3.5 million is quite a step down from that.
Now that might be just fine for Howard Stern, like it might be just fine for Joe Rogan, but I think it's fairly likely that this kind of deal does come down to trading cultural cache, and one's status as a household name for cold hard cash at the end of the day. In that sense, Rogan's move to spotify might be more like a popular touring act being put to pasture as a headliner in Vegas than it represents him taking on the mantle of mainstream media.
I suspect Rogan will see somewhat less of a drop-off than stern since the barrier to entry for Spotify is significantly less than satellite radio, but I suspect we will see that the medium has been a major component of his success: his show is pretty good when it's free and when it's everywhere. In other words, like Howard Stern was the best thing you could find when flipping channels on the way to work, Joe Rogan's content is often worth a click in a sea of mediocre content. Whether people will actively seek it out remains to be seen.