DotA2 actually has some measures in place that lessen the impact of trolls or AKF'ing/quitting players, such as sharing their gold and control among the remaining team, or disabling their abilities that can harm their own allies. In League of Legends, it's a guaranteed defeat if one ally decides to troll or quit (assuming comparable skill on both teams.)
I've always thought that instead of reports and bans, team games should have a peer reputation system, like Reddit or HN's voting.*
MMR/game skills should not be the only criteria for matchmaking; it should include reputation/social skills as well. Oh and disallow global chat in ranked matches.
EDIT: Yes, it makes for a nice thought experiment to come up with a reputation system that would be resistant to "gaming" or abuse, as others have mentioned.
One possibility is to associate a cost to upvoting, and/or make downvoting an earned privilege, similar to how new accounts on HN cannot downvote.
[0] https://www.reddit.com/r/DotA2/comments/46y4v8/dota_2_behavi...
[1] https://www.reddit.com/r/DotA2/comments/351mdk/discussion_do...
In addition, even for non-competitive games, players who obviously troll and are flagged by the community are pushed to a separate pool with other trolls.
The system isn't perfect but appears to work fairly well for new players.
We almost even won a 3v5 once, but we didn't quite close it out fast enough, so they finally caught up enough to take us down. And it was down to the wire.
That was actually one of the better games I remember, just because it was so intense, though that certainly isn't normal.
Then again, there is another side to things. I once had a team where we all completely sucked and we got pounded into the ground... but every single person was still nice and friendly. We all made friends after game, joined into a group, and played a lot of games together for some time after that.
In the very high skill bracket (3.8k+), I've found players on US East and US West are extremely toxic, and the playstyle is relatively boring. So these days, I queue on EU West or EU East.
Especially true, since Dota2 is generally considered less newbie-friendly than LOL.
But I think the general consensus is that LOL is worse.
Both are bad enough that I stopped playing them very quickly. SMITE I stuck with significantly longer.