I've been invited into quite a few exclusive social networks (scenes, clubs, societies, networks, etc.) and they all have a similar dynamic. There is a core group annointed by someone legendary who doesn't really participate much, a senate of old guard who run things in the background, some quirky legacies who are tolerated but marginal, a few social queen bee types who gatekeep and police the tone, and then a lot of social butterflies who promote and attract people into the gateway funnel. It's the model of leader/essentials/influentials/interchangeables model.
What I'd speculate happened (or will) is they didn't have a good funnel because the top influentials were too high stakes, and it creates the incentive for arrivistes to knock the ladder away behind them, which tanks growth. Clubhouse doesn't have enough privacy to invite your friend who might attract tone police attention, and growth depends on being able to select from those very people in a funnel.
Basically, the path from outsider to interchangeable to the more socially secure influential level is too unstable, and strivers won't invite their off-brand friends.
Where I have seen other networks and scenes grow is you need a lobby that is attractive to people, and then once you are on the other side of it, you are securely inside, at least the first level. It requires a layer of real privacy and privilege, and even relative secrecy. Maybe Clubhouse secretly has that now, but it's perhaps too secret, as this article is about how their funnel into their pipeline is drying up.