You realize people have tried that before as well, e.g. Parler, Gab, Truth Social, etc?
You're likely right to some degree though, in so much as that there will be a group or a few who will for reasons (whether truthful or not) don't want to associate with a site owned by Musk and will try to start and create a successful site; it's not easy though - and why Musk is willing to buy a head start for ~$42 billion.
Still, an alternative wouldn't succeed.
Beyond the moderation, these sites have the problem that they are inherently set up as a sort of opposition haven -- "we're twitter for people banned from twitter!"
This results in a phenomenon Hank Green calls the "worst people problem"[0]. I'm not a huge fan of the judgemental term, but the core idea is similar to pointing out the inherent contradiction of, say, an anarchists' society. You end up with a userbase that is defined by its desire to flaunt rules and reject any normative standards.
[0]:https://threadreaderapp.com/thread/1348101443404787718.html
(somewhat related and worth a watch: Folding Ideas' 2017 video essay about YT alternatives. Alt platforms have had inherent problems even before the more recent political bent: https://www.youtube.com/watch?v=r3snVCRo_bI)