>If you don't trust Ledger – how much do you trust your VPN provider? They know both your original IP and what you're doing on the Internet (or at least to which services you're connecting).

Well, the big difference (for what OP is worried about) is that two distinct companies would need to be compromised instead of just the one.