You can use "keyscript=" in "/etc/crypttab" to run any script you want.

Using that you could set up networking and download the key from somewhere (say a remote KMS that would need user approval before continuing). So you would reduce your problem to being prompted on phone to unlock your desktop.

You could even use some sort of hardware key to prove request came from the physical server. (simplest might be a RP2350 with some key burnt in)

Or you could use something like SGX if it's still available anywhere to prove the same.