I can't speak for Dota, but I quit playing league because I got tired of how badly Riot has implemented their ranked system.
There's so much talk about toxicity in league, and very little about what CAUSES the toxicity.
Hands down the biggest issue is not coming down quicker on trolls and AFK's in ranked. They have a system in which they literally FORCE you to play out an hour long game when someone never connects, leaves after 5 minutes, doesn't get what they want so intentionally feed, and so forth.
For someone trying to climb that's automatically 2 hours gone. 1 hour for the game they're forced to play out, and 1 hour for the game they have to win to even out their ranking loss.
I have literally seen streaks of 60-70% of games with one of the above (troll or AFK). Eventually people get frustrated by it and that's when the toxicity starts.
If Riot wants to improve toxicity they need to protect the ranked experience better. I have so many things going on in my life that losing 2 hours because some asshat on the internet decided to troll is just not worth the time. 2 hours is a LOT of time, and while I loved the competitive nature of the game, eventually I just walked away from the game.
Shortly after I decided to walk away from the game, I came across the following video, and it describes EXACTLY why I stopped playing, and where my frustrations with the game where.
https://www.youtube.com/watch?v=5UfO7qOHx5M
The day I quit was when I sat down on a Saturday to spend playing league (one of those rare days where I got to do what I wanted). I spent 2 hours waiting on dodges (where it makes you wait when you leave during champ select). I kept dodging because I kept getting trolls in champ select. And then something like 4/5 of the games I did get into had either an AFK or someone who got pissed off in game at someone and started trolling.
I just thought to myself, what am I doing? There are so many more productive things I could have done today, this game does nothing but frustrate me.
I haven't played a game since, although I still watch LCS.