The fundamental problem with "penumbras" is that once you allow that as a way of reasoning, pretty much anything goes. At which point the Supreme Court becomes an insanely powerful body for something unelected and with a lifetime tenure, and SCOTUS nominations become worthy of scorched earth mode of political warfare.

Thing is, we have the constitutional amendment process for a reason - it is supposed to be a living document, true enough, but it's supposed to be updated through political process, not judicial. The problem, of course, is that the bar for amendments is so high that they become impossible in practice; but it's also impossible to fix that without an amendment. Thus, so long as enough political forces believe that the status quo is preferable to other options, it'll remain that way, and we'll see increasingly contorted Supreme Court decisions to reach goals of whatever party controls it at any given moment.

That leads to another problem, which is that SCOTUS, and the system as a whole, rapidly lose legitimacy. At this point, it already doesn't matter much whether any given decision is legally sound or not - it will be supported or opposed consistently along partisan political lines. The same people will castigate the judges for "inventing" rights in one decision, then blame them for not buying into the rights "invented" by other judges. I don't know how much longer we have left, but this passage from Federalist Papers (#22) comes to mind:

"... two thirds of the people of America could not long be persuaded, upon the credit of artificial distinctions and syllogistic subtleties, to submit their interests to the management and disposal of one third."