Pickle’s Nine Flaws
nedbatchelder.com
nedbatchelder.com
Formally serialization/deserialization is very crunchy and precise. (And I remember how stoked I was to find out that Python included an implementation!) In practice, things get messy and we break the implicit assumptions.
Is it a flaw of the pickle module? Or are our designs too clever?
Patient: "It hurts when I do this."
Doctor: "Don't do that."
;-)
* Old pickles look like old code: Again, convert your object into json and serialize that to your database. Oh no, you are missing an attribute. Pickle should not be used so you don't have to employ a release engineer.
* Implicit: No software works everywhere with defaults. So use copyreg.
* Over-serializes: USE copyreg.
* __init__ isn’t called: USE COPYREG.
* Python only: what's this for, then? http://www.picklingtools.com/
* Unreadable: Great feature.
* Appears to pickle code: Another great feature.
* Slow: check again, it has been 8 years. I can't find any faster method.
???
From the picklingtools FAQ:
What versions of Python does PicklingTools support? Historically, versions 2.1.x to 2.6.x have been tested extensively. More recently, 2.7 has been tested and should work, but it has not been tested as much as the other versions.
3.x has not been tested: We are waiting for our main paying customer to adopt the 3.x series.
There are lots of good use cases for deserializing untrusted data -- it's what you do in almost any client-server situation. So the fact you can't do this with pickle really is an important limitation.
The article is, after all, about reasons not to use pickle. :-)
Yes, and untrusted user data should not have arrived at the system in pickle format. And in case of RoR, it should not have arrived in YAML format or loaded with YAML.load[1] So my point stands.
[1] https://gist.github.com/niklasb/df9dba3097df536820888aeb4de3...
This is one of the few things Java gets right about its built in serialization: if you have an object that can't be serialized, anything using that object has to declare it as transient, meaning it won't be serialized or deserialized. Hopefully you'll think about whether the result makes sense before using the keyword.
If you don't mark an unserializable field transient, you'll get an exception at runtime. It's not enforced by the compiler, which would be ideal, but linters will warn you.
The fact it keeps "old code" is a feature. The object is exactly as it was at the time it was saved.
I wouldn't use it, simply because I like my formats easy to inspect (if you want privacy, just put it on a secure channel like TLS or use some other standard wrapping), but I guess other people have less scruples.
If readability is an issue there's a JSON version that's quite useful.
Other than that, most of the other concerns are addressable. If security matter perhaps use an encryption lib around the pickle, rather than ask for it to be built into it? As for speed, you're already using python and chances are you're not constantly pickling and unpickling?
That still does you no good if you're, say, a server getting data from a client. Very few servers want to allow clients to execute arbitrary code inside them.
That still leaves some situations where it can be used - but it's a major limitation on the scope of those situations.
For numeric data, H5 is nice. For configs, JSON is pretty much a standard. For Python code... well, nothing beats Python code.
I author package "FooBar". FooBar creates objects of type "Baz" which will later be serialized by you, the FooBar user. Within the FooBar package: "def show(baz)" taking a Baz and displaying it. When I decide at one point to add an extra attribute to my "Baz" objects, I cannot (or should be extra careful) to use that attribute within that method if I expect the "Baz" objects to be pickled. This situation can be unpredictable and cause hard to track errors.
On the other hand, this is part of how erlang distributed systems (which is crazy easy) can communicate with extremely simple semantics, and the security model for erlang distribution is very explicitly "trusted only; locking down the cluster as a single security domain is YOUR responsibilty".