The interviewing process would usually flow along a line of screening out candidates that are potential hires and THEN trolling their LinkedIn profiles to see who you both know in common. I think it would be tiresome to do that for EVERY candidate so that you can ask them during the interview "Mind if I contact Foo.".
As long you apply a little discretion about it I don't see what the big deal is. EG: don't contact someone at the candidates current company unless that contact really IS you BFF and you know they can keep the conversation private.
Also, if you do get any odd reviews you'd probably also want to give the candidate room to explain. I wouldn't have a big issue with someone contacting random ex-coworkers of mine as references. There is one person in particular though that probably wouldn't have good things to say, but digging further on that guy you'd realize he's a complete nutjob. So, you'd just want to make sure you properly tempered all the feedback, but in that scenario you should see overall trends and be able to spot outliers.
I guess I'm also not highly opposed to this because I think most current interview tactics suck as screening methods, and everyone hand-picks their references anyway, so you kind of need to go out of band to get a really reliable reference.
I'll also add that I think it depends somewhat on the level of person you're hiring. I don't know if Halligan still personally interviews everyone coming into HubSpot. If he does, I'm pretty sure he is not the one checking references for a level-1 phone jockey. So, we're talking mostly about the higher-order candidates, key individual contributors and managers that he's doing this for. It just doesn't strike me as such a deplorable practice.