There might (in theory) be some kind of backdoor on the device, that could allow the chrome app to send the private key or seed somewhere...
Basically, unless you're 100% vigilant literally all of the time, you're in trouble. This is just one reason why we invented banks, to centralize risk and mitigation.