But your point about this slippery slope is well taken.
Thomas Jefferson had this to say about it, per Wikipedia:
"You seem to consider the judges as the ultimate arbiters of all constitutional questions; a very dangerous doctrine indeed, and one which would place us under the despotism of an oligarchy. Our judges are as honest as other men, and not more so. They have, with others, the same passions for party, for power, and the privilege of their corps.... Their power [is] the more dangerous as they are in office for life, and not responsible, as the other functionaries are, to the elective control. The Constitution has erected no such single tribunal, knowing that to whatever hands confided, with the corruptions of time and party, its members would become despots. It has more wisely made all the departments co-equal and co-sovereign within themselves."
The subsequent history has shown he was spot on. Given that the Switch In Time That Saved Nine waved through FDR's New Deal, and most subsequent gross violations of the Constitution, shows in practice they're really bad at what they arrogated to themselves, and they've far too often given the other branches of government cover. E.g. "sure, this might be unconstitutional, the Supreme Court will decide!" Fairly often the "might" is really "is", and later politicians who depending on a saving throw from the Supremes got disappointed. E.g. McCain-Feingold.