It's very hard to not spend all of your time managing people.
We like think that even responding to issues and pull requests are "just code", but they are actually people problems. You're managing people.
For example, this person spent a week on this pull request, but it wasn't something you wanted to be done, and they didn't do it the way that it should have been done. What do you tell them and how do you tell them? Now let's say they are a vocal person in your tiny ecosystem. It's not easy.
And a lot of the ecosystem you get is pure luck of the draw. Since ecosystems self-select (people who don't fit in in certain ways move on) you can get dynamics that really have nothing to do with you, whether you had zero presence or were 100% prescriptive about how the ecosystem should work. It's kind a bizarre to see over time. (Message boards back in the day were a good example of this too)