I would love to build a community of developers and contributors on GitHub around my project - who doesn't want to be a Benevolent Dictator for Life? But I can relate to how you feel - on the one occasion when someone created a PR contribution my reaction was to panic, and then copy the (very minor) changes manually over to my current working branch and merge that way.
I expect I've (accidentally? subconsciously?) built barriers to contributions over the years. An idiosyncratic coding style probably makes it more difficult for others to figure out what's going on in the code - especially as I've never written a coding style guide for the project. The rudimentary toolchain around the project - in particular the lack of unit tests to give others confidence in their PR's quality - certainly doesn't help. Writing a Contributor's guide has been on my to-do list for at least 5 years; it's so easy for me to find an excuse to do the work "tomorrow" (which never comes.
And the stupid thing is that this lack of collaboration seriously damages my project. When devs look around for a canvas library to add to their projects, they care a lot about the community surrounding that library - why risk going with a solo-dev library where there's no guarantee that the library won't be abandoned tomorrow? I honestly don't blame them for choosing to go with the competition ... however sad it makes me feel.