They certainly would not be more accountable if their appointments were more aristocratic in character (i.e. they inherited them from their ancestors who had been appointed for arbitrary reasons by an unelected monarch) and, like aristocrats, they didn't have to explain how and why they exercised the powers given to them.
The "easier" route is for the legislature to "see" the ruling and pass a law that counter-acts it. The general public can _also_ call their own Constitutional conventions and pass amendments to the Constitution to hold SCotUS responsible.
The real "problem" is all of these solutions, by design, require significant effort. If you want changes, you will need to make them happen.
A political mechanism defined on paper does not mean anyone will ever invoke it.