The whole point of offline storage is to guard against the risk of whatever internet-connected device you use to store your keys getting compromised.

If you make a mistake or fall victim to an attack that lets someone steal the encrypted file containing your key, there's a good chance the attacker will also be able to install a keylogger and get your passphrase.