Most instances will quickly shut down any spam, so it doesn't really fill the federated timeline that much. Local accounts are something you as an administrator need to act on, so they are the annoying ones.
In my experience, instances that allow spam are rare and the get quickly blacklisted by everyone from showing up in the timelines (there is luckily a neat middleground between blocking the instance entirely and not doing that; allow local users to subscribe to that instance, which is my default action for all instances that aren't hosting illegal material for me).
Good thing is that the Mastodon instance owners use shared block lists for servers that are currently used to ban free speech instances; that infrastructure could also be used to create a collectively managed list of instances that are full of spam.
Traditional tools like muting a user, silencing an instance (if they're spamming up the Federated Timeline on other instances, like Humblr.social does) and blocking (if the mods are total shit) can still be used. Small instances are the bulk of the content, so you aren't about to block them. Large instances like Mastodon.social are generally blocked tho (if your admin/democracy cares about a vibrant Fediverse).
Ah, so you achieve vibrancy by .. blocking a large number of the users?
It's not so much about "blocking" mastodon.social as not choosing to link up with it - because your instance is a specific community, and actual human social relations thrive when they can choose who they're hanging out with. So you choose to talk to other smaller, more diverse communities that you can actually get to know a good chunk of the people in them, instead of just drinking straight from the firehose that is a big instance.
Federating with another instance is basically two groups of people deciding to hang out together, effectively merging. Federating with mastodon.social means, effectively, merging your community with mastodon.social ... which means submerging it beneath the mass of mastodon.social's much greater activity.
A vibrant Mastodon isn't a more-obtuse recreation of the Twitter experience. Twitter is already Twitter. It's about fostering all the things that Twitter isn't - personal and Dunbar-scaled and locally moderated and diverse.
Also, I think Mastodon’s design deliberately steers instances away from becoming large ones; it rather tries to be a nurturer for various small communities and subcultures. It doesn’t view size as a good thing, and that’s totally okay. Nowadays, I think we need less monoculture and more diversity/creativity in our society.
Might be your particular corner of the fediverse, I prefer to have as much reach as possible.
You could have a personal (stored on device) Bayesian filter that learns from your preferences, and a shared one to block spam spotted by your contacts
As other people suggested, I don't think it should be at the node operator level, for freedom of speech concerns