Well yeah, but this assumes I don't know the fact the physical access had place and proceed to enter the passphrase revealing the key unawaringly afterwards. If this is what "having physical access" means - then yes, I agree, game over, and ORWL solves it.

But if I know the attack took place (FBI broke into my house, the computer is locked in a safe etc.) - the data should be secure.