Life is too short to waste your free time being somewhere that you don't want to be.
(It's also easier to change a "no" to a "yes" later, then to change from a "yes" to a "no".)
Then again, you might be surprised at who you meet when you're somewhere you don't want to be.
Too many people are shut-ins and never meet anyone, so never get the chance to meet someone who might change their life, help, or stimulate them.
I'd err on the side of meeting more people and giving them a chance, unless you're already a social butterfly with way too little time for yourself or the people you care about.