>What is the mathematical formula for a valid court order?
This is a political / social issue. A human has to decide.
>How does it look different from a court order signed by a judge held at gunpoint?
It doesn't. You could similarly hold developers at gunpoint to push malicous updates too or at users to give them their messages. Putting people at gunpoint is illegal.
>How does it look different from a court order signed by a dictator's minion?
Judges could have their own private key so it would be noticeable that the minions private key would not be trusted.
>How does it prevent someone from tricking a judge into signing an order to decrypt message 2421425241 instead of 2421475241?
It doesn't. This is solved by laws that make such an action illegal to do
>What is stopping the service from accepting invalid court orders?
They want to protect the privacy of their users so they should reject invalid court orders.
>What is stopping the service from just decrypting everything for convenience?
They would not have the keys LE has. So it could be setup such that LE's keys are required to decrypt something.