Smaller communities tend to be more familiar - people recognize each others names, and people have each other's backs, when someone is trolling or something. When things grow beyond a certain size - maybe related to Dunbar's number - the group dynamics change.
In some cases this can lead to the community to loosing coherence or even becoming toxic - but I don't think that's inevitable. I believe it IS possible to grow a community to a larger size and have it stay nice. I think though, that in order to keep a community nice when growing beyond a certain point, some effort is required - effort which wasn't necessary while the group was smaller and kinda self-organized.
Also, growing slowly rather than quickly might help - as newcomers have some time to acclimatize and internalize those unspoken social rules that govern a group's dynamic - and will be able to pass on the knowledge to those coming after. People who join a group will eventually adapt to the group - but that process needs a bit of time. But when growth is too fast, a newcomer might have more interactions with other newcomers rather than long-standing community members, and thus will adapt to something quite different.