Homeschooled kids are a minority, and therefore formerly-homeschooled adults are a minority too. More of a minority, even (since the rate of homeschooling was lower when we were growing up). It's not "telling", it's simply the fact that not a ton of people exist to give a defense.
For my part I was homeschooled and had a great experience. My parents were able to challenge me in ways that the local public school wasn't willing to, and they made consistent efforts to make sure I had social exposure to other kids. I don't believe I am lacking in any way because my parents homeschooled me - in fact, if anything I believe it was the better option for me.
And I think you need to remember that the other side of the coin exists too. My wife went to public school, and was bullied by other kids. She has social anxiety and other issues to this day (she's 42 years old) as a result. That doesn't mean public schools are all bad, of course, but people need to stop ignoring the downsides of public school when they compare the two paradigms.