While this argument can and have indeed happened in other instances, this is akin to saying that we should not give anyone any powers to do anything because it is a slippery slope that they can use it to do bad things.
What then sets out the difference between what a slippery slope and a non-slippery one is? Checks and balances and the history of USA have shown that this is indeed what can reign in the worst instincts of any single entity. History of course have also shown when these failed and these should serve not as a reason to reject the idea of checks and balances but as acknowledging it's imperfection and think of ways to mitigate it.