I think this is inherently a sociological problem than a technological one. When there are enough people in an organically grown group, the community will become unsustainable and then 1) it disintegrates into multiple groups until it becomes stable again or 2) it transforms into a hierarchial, authoritative society where a small group of people makes most of the decisions.
Also, I think Mastodon’s design deliberately steers instances away from becoming large ones; it rather tries to be a nurturer for various small communities and subcultures. It doesn’t view size as a good thing, and that’s totally okay. Nowadays, I think we need less monoculture and more diversity/creativity in our society.