Users are the content on social networks. If you charge them to join, you will have fewer users and less content on the network. That reduces the value so you can't charge as much, and the cycle continues.
I don't think you're entirely wrong, mind you, ad supported services are the default and any competing service that requires payment operates at a disadvantage, which is why I continuously come back to the idea that surveillance capitalism needs to be regulated. On a recent episode of Better Offline, the host interviewed an economist who has suggested a steep tax on digital ad spend, specifically to open the space to alternatives away from people giving away their privacy. I think that's a great idea, not as extreme as my ideal solution which would be the utter dismantling of surveillance capitalism entirely, but still.
The problem is money is literally the power here, as it is in most places, and the users are contributing none, and the advertisers are contributing all, and this is basically the source of every cancerous, inhumane and exploitative problem you can think of in social media. And you are 100% correct: a paid instagram will never succeed as long as instagram remains free. Then, IMO, the logical next step is we need to make it not free, or at the very least, take up some of that profit and put it towards other ends that we can actually affect change on.
"The market" is clearly incapable of solving social media, it only has made it worse, ever, ever worse. We need something else.
Have you looked into ATProto? (by Bluesky)
1. It separates the data, app, algo, and moderation into pluggable modules
2. You can build any app on top of it, all app user data ends up in the same user database (sqlite db per user)
3. They supposedly have a non-ad monetization plan, but haven't publicly announced what it is yet