Don't Pickle Your Data | Hacker News Reader

Don't Pickle Your Data

benfrederickson.com

JSON, while a fantastic lightweight data format, is insufficient for storing even moderately complex objects. It can not properly differentiate between tuples and lists, it can only accept strings for object keys, and it only stores unicode strings.

You also have to write custom (de)serializers if you want to store datetime objects (my personal pet peeve), any of the special python containers... basically any time you want anything other than an object, array, string or number.

Unless I'm dealing with untrusted data sources, or need to interoperate with other languages, I will keep using Pickle.

I built an extensible type system on top of JSON for serialization and validation. Among other types, it ships with DateTime (it represents dates as ISO strings). It also lets you define your own types, as simple or complex as you need them to be. If you've got a minute to take a look, I would really appreciate some feedback :)

http://www.cosmic-api.com/docs/teleport/python/latest/

Heh. Pickle is not really sufficient for some edge cases, too. Luckily, there's Dill that can take almost perfect snapshots of the whole interpreter: https://pypi.python.org/pypi/dill

Also Dill's developer Michael Mckerns is super responsive and helpful. Hopefully someday pickle will be replaced / augmented with dill in the standard distribution.

Also, most critically, JSON doesn't allow circular references.

"Use pickle with caution" would be better advice. There are plenty of cases where your data is under control and pickle is a huge time saver. It takes work to translate most data structures to/from json.

Your benchmarks don't mean much because you're giving different data to different packers.

cPickle gets a list of objects but json gets a list of dictionaries? The cost of converting the objects into the dictionaries is conveniently excluded from the json benchmark.

You should try serializing the same list of dictionaries, use the highest pickle protocol and repost results.

This benchmark doesn't specify a pickle protocol, which forces Python to use a big, inefficient format.

I filed a bug. https://github.com/benfred/bens-blog-code/issues/1

This is a huge thing he missed. Anyone using pickle for non-trivial work is doing cPickle with PROTOCOL_HIGHEST.

I also went and tested it.

Baseline JSON:

    --------------------------------------------------------------------------------
    JSON
    packTime 0.612724065781 s -  163205.601975 items/s
    unpackTime 0.782174110413 s -  127848.772631 items/s
    size 174.26637

Baseline cPickle, ascii protocol:

    --------------------------------------------------------------------------------
    cPickle
    packTime 2.41442704201 s -  41417.6938297 items/s
    unpackTime 0.875658035278 s -  114199.831408 items/s
    size 286.26637

cPickle, highest protocol:

    --------------------------------------------------------------------------------
    cPickleHP
    packTime 1.02942800522 s -  97141.3245929 items/s
    unpackTime 0.583297967911 s -  171438.965162 items/s
    size 198.26637

For giggles, I evened the playing field and let cPickle at the same data structure as JSON, ascii protocol and highest protocol:

    --------------------------------------------------------------------------------
    cPickleJsonData
    packTime 0.642832040787 s -  155561.629874 items/s
    unpackTime 0.478959083557 s -  208786.101847 items/s
    size 205.53356


    --------------------------------------------------------------------------------
    cPickleHPjsonData
    packTime 0.285845041275 s -  349839.897708 items/s
    unpackTime 0.340456962585 s -  293722.881273 items/s
    size 175.26637

My conclusions? Serializing dictionaries is easier than serializing objects, and using a non-ascii protocol offers some obvious benefits. JSON is not obviously better than cPickle when comparing apples to apples.

It's also worth noting that comparing the json module to the pure python Pickle module isn't a fair fight either; json is at least partially written in C.

I wonder how ujson (pure-C) compares? https://pypi.python.org/pypi/ujson

At the end of the post he says that only Python can parse Pickle. This is not really true. You can write a Pickle parsing library in any language (maybe some special use would not be possible).

For instance I have implemented enough of it[0] to use in a drop-in replacement for Graphite[1].

[0]: https://github.com/noteed/python-pickle [1]: http://graphite.wikidot.com/

  >>> import json
  >>> json.dumps(set())
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
      return _default_encoder.encode(obj)
    File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
      chunks = self.iterencode(o, _one_shot=True)
    File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
      return _iterencode(o, 0)
    File "/usr/lib/python2.7/json/encoder.py", line 178, in default
      raise TypeError(repr(o) + " is not JSON serializable")
  TypeError: set([]) is not JSON serializable
  >>> from decimal import Decimal
  >>> json.dumps(Decimal(1))
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
    File "/usr/lib/python2.7/json/__init__.py", line 231, in dumps
      return _default_encoder.encode(obj)
    File "/usr/lib/python2.7/json/encoder.py", line 201, in encode
      chunks = self.iterencode(o, _one_shot=True)
    File "/usr/lib/python2.7/json/encoder.py", line 264, in iterencode
      return _iterencode(o, 0)
    File "/usr/lib/python2.7/json/encoder.py", line 178, in default
      raise TypeError(repr(o) + " is not JSON serializable")
  TypeError: Decimal('1') is not JSON serializable

What, precisely, is your point? The article points out JSON's limitations.

I like y_serial, a library to store compressed Python objects in a sqlite database.

http://yserial.sf.net/

> Given the downsides though, its worth writing the little bit of code necessary to convert your objects to a JSON-able form if your code is ever going to be used by people other than yourself.

Disagree strongly. You have no idea how complex my graphs of (pretty damn complex) python objects are and how much I value being able to change their organization without having to rewrite the serialization code every time I do so.

Don't fear pickle. Just don't expect it to be secure or amazingly fast.

Pickle seems to be five times faster than standard json. If I store lot of data and want it to be compact then I'll use xz on top.

In case anyone doesn't know, Python has had a json library since version 2.6.

That's not the issue. The issue is that python objects are not serializable into json without writing a custom serializer. Rather than telling devs not to use pickle, why not build a json serializer that can handle simple objects and try to get it adopted into the python standard library?

I thought pickle output was just JSON? Shows what I know.

If it was JSON, how would you pickle dictionaries with tuples as keys?

The same way you would in JavaScript: turn it into a string first. "(1, 2)" Its not that serializing python objects to json is impossible, just that there is not a straightforward implementation that everyone agrees is correct.

For a fully general python <-> json serializer/deserializer, you'd need a bunch of extra annotations etc that would make it look very weird compared with what most people expect when they think of json.

Reply on news.ycombinator.com