Should the executive branch agency make decisions by fiat, without due process? The use of due process doesn't make them a judiciary; it's just a more fair way to make decisions within their purview.
Should we move every such judgment out of the executive branch's hands? First, they have to be able to make some decisions.
But the heart of this issue is, as is well known: There aren't nearly enough capacity in the judicial branch to handle all that work. It's a trick to prevent lawful and needed regulation from happening by severely cutting their capacity.