I think the idea is that the seed and associated private keys are stored securely on a secure element, and they cannot be transferred to your computer. Only a signed message that allows broadcasting of a signed message can. But I could be mistaken?

Is this just a technical possibility or is there evidence that this could be the case?

You don't even need that, there's been a bunch of fraud in hardware wallets from people simply pre-setting the seed and shipping them with a "this is your seed, don't forget it" slip. Not everyone will use that seed, but some do, and then they lose their money.

Basically, unless you're 100% vigilant literally all of the time, you're in trouble. This is just one reason why we invented banks, to centralize risk and mitigation.

The chrome app doesn't know the private key and the nano itself only runs signed code unless you imply that the Ledger Company itself planted the backdoor.