Hacker News Reader

Comment by ivankirigin | Hacker News Reader

This is some really useful stuff. I learned a lot of this over time, and wished I had known it sooner.

The point about schema evolution is important.

I've taken to using a pickled python object in a text field in every model. That flexibility to store arbitrary data in a hash is amazing. I've used it to, for example, represent twitter accounts that are oauthed, with password, following the business account, or just nothing special. 4 kinds of accounts, with logic in the model to deal with it properly, all without evolving the schema. I only started with the first 2, and added the latter two without a problem.

I'm pushing this a bit further now, and transitioning away from models altogether, and experimenting with a non-relational database like couchdb. I'm debating moving to google app engine too, but found some details (like getting on your own domain) to be pretty annoying.

View on HN

Really don't like the sound of that, pickling allows you to serialize any python code, and consequently deserialize it. Presumably you are placing user submitted data into this pickle and therefore without careful sanity checking wouldn't this potentially allow the user to inject code which may be run on deserialization?

Someone else may be able to shed more light on this and correct me (please do!) but thats my concern and would love to hear if it was a valid one or not.

It is possible to build a poisonous pickle which, on depickling, calls arbitrary standard library functions, so accepting pickles from untrusted source is a No.

One may think that if only the internal model of the user input is pickled, and the user only gets to update this internal model through web requests in defined ways, thus never getting to upload pickles or construct arbitrary complicated python objects, it should be relatively safe.

However, what remains is that if someone manages to blindly inject SQL, he can inject executable code into your application, instead of just injecting data.

Whoa! I was assuming this worked like doing a JSON grab-bag column in Rails or J2EE, which I am a fan of. Executable code would make me much, muuuuuuuuuch cooler on the idea.

Can you just do the same thing with JSON, Yaml, or your favorite serialization method of choice that does not have catastrophic consequences if parsing gets subverted?

> Can you just do the same thing with JSON, Yaml, or your favorite serialization method of choice that does not have catastrophic consequences if parsing gets subverted?

Sure. You can even extend the `json` module to serialize custom data types not defined in the JSON spec.

No, I'm just serializing dicts. Any user input is already in a string, so sanitized. I also usually make helper functions to get/set things in the dict, so I don't operate on the field outside the model often.

This snippet is really helpful: http://www.djangosnippets.org/snippets/513/

Come to think of it, I think Hacker News works like this.

When you upvote and need to sign in, the code to execute the upvote is stored to be executed once you sign in.

Pickling python objects and storing is an very old practice in Python world. Zope has been persisting objects in ZODB (a giant pickle jar with some transactional features) since 1998.

New to me :)

I love it

> I've taken to using a pickled python object in a text field in every model.

That's basically what App Engine does right there. It stores by an id, then the value is your object serialized as a protocol buffer.

The App Engine datastore allows you to look up by other properties than id because it indexes every field of your object in the background without you having to deal with it. You can query by pair-wise or more-wise indexes too, but you have to create those manually.

Storing pickled python objects is fine as long as you don't care about relational data. But if you aren't careful, data integrity will probably suffer over time as the complexity of your application increases. And that is not good. Anyway I'm sure a lot of people here know this stuff already. Just do your research and make sure the route you go makes sense for your application and your scaling requirements.

Would be interested to hear your take on Django+CouchDB - are you using a library like Couchdbkit, or rolling your own? Are you connecting it to the built-in admin?

Still exploring. I'll post something when it actually works.

Reply on news.ycombinator.com