Also to your point: "can't we just encrypt it?"
Its someone else's computer. The TPM is controlled by someone else. You can't really process on a machine that has a compromised urandom/TPM
Also the bigger issue is having all your access revoked over night. Thats the bigger fear.