The key isn't in your head: you know the passphrase used to decrypt the key which is then kept in system memory. Techniques like the cold boot attack[0] or row hammer[1] can be used to retrieve the key and access your data. In the case of non-hardware-TPM secured encryption schemes the kernel or bootloader which must remain unencrypted for the system to boot can be backdoored to record the passphrase and/or the key.