When it's "cold" (offline, or running but in system-low mode), you have no real protection of the OS, but you essentially distrust the machine whenever it gets rebooted without your consent. If someone can get root on the server while it's in "high mode" (normal operation, after keys are entered), any protections are irrelevant anyway.
The risk is if the attacker can cause enough reboots that you stop caring about reboots so much, and just blindly enter the passphrase; the attacker then puts up something which steals the passphrase, and then uses the passphrase on an already-imaged drive, winning the game. Or if the attacker can use one of a large number of vulnerabilities to get access to your data when it's running in system-high mode; either an app-level vulnerability or either local access + local root, or remote root.
Server technology sucks for anything needing great security but adminned strictly remotely.