If he can run a keylogger on whatever computer you use to unlock your wallet, then it does it really matter whether you store it offline on USB stick in a safe or not.
That's where hardware wallets like Trezor or KeepKey come into play. Any transactions are signed on the device and sent back to the computer; your private keys never leave the device.