Actually, the U.S. Supreme Court has a formal definition of freedom: "The right to be left alone." This is a very frequently quoted phrase in USSC decisions. BTW: the right to privacy is considered by many to be half of this.
The phrase you quote is most associated with Louis Brandeis' conception of a right to privacy (it was his in-a-nutshell definition).
Privacy and freedom are not the same, obviously. And, there is nothing explicitly in the Constitution (or the Amendments) about privacy per se, which is why Brandeis had to write the article grappling with the issue.
[citation needed]