The problem is that it's an awful lot of work for everyone to decide for themselves, and I don't think the average user is going to care enough to build their own moderation algorithm. They'll just complain and move on
This is objectively false. The forums run by a tinpot dictator who excludes too many things that people want to see have zero users, not the most users. Many of the most popular forums became popular before they had any kind of strong moderation and only adopted it after they already had a lot of users. HN is one of the rare examples where the moderation was present from the beginning, a major reason it works is that it's a specialized site that doesn't target general audiences, and it's "popular" but not half as popular as 4chan.
As a general rule people don't like spam, but if the place where their friends are has some spam, they're going to stay there while complaining about the spam. And, of course, they hate it when somebody excludes something that they want. But that might not get them to leave either, because network effects are powerful, even when they hate it. Twitter users have been calling it "the hellsite" for many years while continuing to be Twitter users.
Which is why large general-purpose platforms need different algorithms for different people -- because different people want different things.
There can be one filter that only blocks spam and scams, another that blocks conservatives, another that blocks spam, scams and celebrity gossip, another that only shows technical content etc. You open the config options and pick one, like you choose an ad blocker or a radio station. If you don't like it you pick a different one. It shouldn't be that hard.
And if the answer is "AI", then who trains it and with what data?
Whoever makes it? The entire premise is that anybody could make one, however they want, and people could choose the one that works best for them. Go create one that operates by manual curation if you want to.
But here's an answer to this one:
> And if the answer is "AI", then who trains it and with what data?
You do. On a network where votes/ratings are public, make a filter that looks at how you vote, compares it to other users to find the users with the most similar tastes, and then shows you the things that they liked. Occasionally it shows you something generally popular or vaguely controversial at random to see if you like it and if you do it shows you more.
This isn't even modern "AI", it's old school machine learning and statistics. And it works pretty well until bots figure out how to game it, and then you have to come up with a way to detect the bots, and it becomes a cat and mouse game.
Which is why we need the evolutionary pressure on the other side. Have lots of different filter algorithms made by different people so the bots have to hit multiple targets at once, and users can pick the ones that are most effective at excluding them and don't get stuck with a crappy one they can't change because it's tied to the network itself.
> Bob wants to open up his feed and see all the content he likes from Alice and Dave and Erin and Frank. If Alice goes somewhere else, Bob can follow her, but then Dave and Erin and Frank are still on the old site, and maybe Bob doesn't want to give that up, but the others don't even know that Alice exists and have no incentive to move.
Why is it objectionable to you for Bob to have the ability to sort his feed differently than you?