I dispute your assumption that the courts need to be able to compel behaviour. At best they need to be able to compel inaction, but they should never be able to compel action
I’m no expert here and have not thought about this extensively, so I have no strong opinion here. That said, at first glance:
* It seems reasonable that courts need some power to compel people to cooperate with investigations — otherwise they’d be unable to act on certain kinds of crimes, conspiracies, etc.
* What alternate justice system would you suggest specifically, which is presumably more fair and effective that this one?
As for acting on certain kinds of crimes, the job of prosecutors is to gather evidence, and find people willing to testify, to support their case. They already do this, sometimes offering things in exchange for testimony (such as immunity for their part in the crime), to get witnesses to cooperate and offer testimony that can be believed. If they can't get someone to voluntarily testify, however, it seems counterproductive to me to toss them in jail to punish them for not cooperating; from the outside, that to me makes it look like their case isn't very good to begin with.
What more do they need? Compelling testimony from an uncooperative witness isn't necessary for a case, nor is it even very helpful: such a witness can't be considered reliable or trustworthy.
In your world guilty parties walk because the state has no power to make anyone do anything. Everything is based upon the prosecutor being Sherlock Holmes and always finding exactly what they need. The real world doesn't work that way.
Are the italicized qualifiers intended to mean something other than that?
If you're on trial for murder, the prosecution would never call on you to testify. Why would they? Of course you're going to claim that you didn't do it. Your testimony would be useless.
Murderers are tried with physical evidence linking them to the crime, and witness testimony from other people. Murderers don't even need to be present to be convicted; this happens sometimes if they're considered a danger in the courtroom, or have fled.
From a quick search, it appears that a trial in the US cannot begin without the defendant present, although it may proceed if they leave afterwards depending on circumstances.