You have to trust the hardware (open source hardware/firmware will be an improvement), and verify addresses out of band (see https://medium.com/@octskyward/why-you-think-the-pki-sucks-b... for solutions), but otherwise you can safely use it on even a malware infested computer.
As a bonus, it can be protected by a pin (with firmware-enforced rate limits). And backups of the seed can be made in case the hardware wallet is stolen/lost/destroyed. They can even be split up using Shamir's Secret Sharing.
(Eventually mobile devices will be able to accomplish the same thing. You could imagine a more advanced version of Apple's "Secure Enclave" that presents the user with a UI (rendered by the Secure Enclave coprocessor) to authorize transactions)