The attacker only needs to have compromised the device which spends from the wallet file
You would need both halves to sign a transaction
At some point during the spend from the wallet, the privkey that matches the wallet pubkey has to touch memory. This privkey can in theory be compromised in a number of ways with malware on the spending system (keylogger, screen caps, process memdump, etc).
I think the safest way to go about this is to generate an entirely new keypair/wallet on an isolated system. Spend from your wallet then transfer the balance to the newly created wallet. This minimizes losses as a result of privkey compromise (unless of course your isolated system isn't so secure)