Congress passes laws, the agencies implement them, if you disagree you go to court. All it's saying is that you don't have to go to the Supremes to get your disagreement to win.
Imagine a patent system where the judge could never throw out a patent, because the experts at the agency (patent office) had granted it, so it must be valid.
In both reality and your supposed system, someone could always go to appointed officials and get their patent enforced - just look at how the Eastern District of Texas operated for decades. In reality, we see far more frequent turnover and shifting of opinion in federal agencies than we do in the judiciary. Just look at how often Net Neutrality has flopped back and forth at the FCC. (Constant reversal of regulatory decisions is also an issue, but it goes to show that the idea that a patent system that exists outside of judiciary control wouldn't have plenty of opportunity to make your case to sympathetic ears is silly)