The GP's point is if you're doing code deserialization, the definition of security is different. The data format having RCE bugs won't be as much of a concern, while trusting the data source will be much more of a concern.

Why do you think the attacker has the ability to change your local files? That's not the attack surface. The attack surface is when it's used in a wire format for network communication. The attack surface is the attacker convincing you to download a file in a format that you don't realize is executable and every file format that includes pickled objects is vulnerable. The attack surface is that you've confused data and code; There's a reason why OpenBSD now enforces W^X, and that is that there's very different domains of trust. I'd trust my neighbor down the street to send me a spreadsheet, I'd not trust him to send me a program (which is precisely why Office Macro viruses were such a problem, because speadsheets could contain programs.)

No, pickling and unpickling files in your user's data-dir isn't a big deal, until it is because your users wanted to share their data online. They'll sync their data-folder with Dropbox, and then their account gets compromised, and the next time they launch your program they've got a virus. They'll download a 'completed savegame' from a sketchy site, and now they've got a virus. They'll get a phishing e-mail, but it's not one of those zips or exes or whatever, it's a file-format they know is your program, and surely that's safe, right?

Don't make your file formats insecure. Don't pickle.