That isn't to say I don't have close friends who do add value to my life, those are the people who are worth spending time on. I don't think its about being a hermit, its about being more selective.
I always try to help people when they ask me for help, even if it has near zero value for me (like helping out relatives with PC trouble). Yes it wastes my time, but at least i could do something good for someone else.
I have a wife and kids, and while I will often go out of my way to go help folks, I'm limited in what I can do... so while I don't mind helping my buddy when his car needs a jump, I have gotten a lot more willing to just not do things like that, especially for folks who have demonstrated that they are selfish actors. I am not down to "one strike and you're out" yet, but at the same time, I have started to feel the pain of thinking "damn, I could be hanging out with my kids instead of taking this moron to get his tire patched."
Neither of those things is especially productive, but we all make choices, and it is healthy to consider alternatives to what we usually do.
There's a great skit on Inside Amy Schumer that describes these people amazingly. Amy and a date are at a sandwich shop near Ground Zero in Manhattan, and her date makes conversation by talking about his experience on 9/11. Amy, however, is completely disinterested and keeps derailing the conversation because she becomes obsessive about whether or not she should ask the employees to put mayo on her sandwich, then further disrupts the conversation by yelling to the employees and throwing a fit when they end up forgetting to put mayo on the sandwich. [1]
I feel like we all have these people in our lives that we stay close to out of guilt, politeness, etc. My mom calls them vampires; they're people who'll cling to you and suck the energy right out of you.