I would call it a concern rather than a problem. It'll be interesting, and likely frustrating, to see what what the Supreme Court does but it is worth noting some potential upsides - for both parties to a case- of this system.
The reasoning for the internal administrative courts is because these (SEC, NLRA, etc.) are highly esoteric and detailed areas of the law. Having people be judges in these cases who are experts can be a positive so they don't get BS'ed and can resolve issues efficiently and effectively and consistently. That is very different from the experience of dealing with federal courts at large where a judge may be hearing a murder trial one day and a shareholder lawsuit the next. Those courts ALREADY have huge backlogs so adding this case load to them is likely to be disastrous as well as resulting in more manifestly unfair outcomes when one side is able to delay to the point where remedies aren't meaningful. The argument that they are political (your overturning) also applies pretty damn well to EVERY court these days, not just internal administrative ones.
I think a useful analogy is to how frustrated much of HN gets every time federal judges struggle to grok technology - whether it be the internet, software code, or privacy. Imagine if there was some sort of court dedicated to dealing with 'technology stuff' where the judges had a background in the relevant areas to make sure reality won the day. There is a place for generalists and a place for specialists. Maybe this isn't the best way to handle it but it is a way.