Stop passing naked dictionaries around: Introducing DStruct for Python
github.com
github.com
The problem with these styles of library is that their modularity is relatively poor. Once you start using attribute access for simple data, or relying on extended features of the container, all your callers, and future reuses of the code are forced to also use that style of access.
While it is easy to mock up a container sporting attributes that can be substituted in place of the instance type, e.g.:
class Bag(dict):
__setattribute__ = dict.__setitem__
__getattr__ = dict.__getitem__
For these reasons, mantra 3 of 'import this' comes to mind:
Simple is better than complex.
I've recently picked up DataKit work again. The local development version has a dict-like object whose only extension is to ensure the type of assigned items matches that of the schema type. This replaces an old 'Entity' class that required attribute access (essentially chosen only because it was "prettier").
A hidden benefit of passing a simple type is easier testability: from a unit test, initializing a class with a dict literal is much easier than needing some attribute-like stub class that must be declared and initialized first (to avoid that, you're forced to test your container class at the same time as whatever caller code is currently under test, which is a broken habit).
As if it wasn't clear: please, please continue passing naked dictionaries around, they're a universal container protocol evolved over 20+ years that wasn't in need of fixing.
https://gist.github.com/24651d78d3f1e3dec6f4 :
def new_dict(args, *kwargs):
"""
Create new dicts
>>> new_dict({1:2}, {2:5}, foo=5)
{1: 2, 2: 5, 'foo': 5}
"""
d = {}
for arg in args:
d.update(arg)
d.update(kwargs)
return d
(essentially chosen only because it was "prettier").
That was my reason as well. It is easier/prettier to write a.b.c. instead of ['a']['b']['c']. But that's all.
But I have frequently seen them being used when class instances would be better suited.
Dictionaries' values cannot be replaced with methods via @property, and as a result, I've found they often need to be replaced with objects at some point. DStruct is a response to that problem where your access interfaces aren't forwards-compatible with the great majority of types you might choose in the future (ORM/ODM instances are a big one).
But I've also found it quite useful in the other direction: when replacing what was once an ORM/ODM class with something mocked (e.g. the data source that was once a database has become a remote resource, local file, parser output, or configuration dict), having to run around changing attribute accessors to dict accessors is a drag at best, and impossible at worst (e.g. If you were using @property extensively to return computed attributes, or you know at hydration time that code elsewhere relies on a schema (I do both a times, as anyone writing large, maintainable applications should.).
As another commenter also mentioned, my approach does yield prettier code.
While it's not always sufficient cause, prettier code is in fact a good thing.
Moreover, it has dependencies with dorkitude_utils and python_memoize, both of them I don't have in my python install.
In the abstract, tuples simply arent suites to all problems (any more than are dicts): order doesn't always matter, and immutability certainly is not always desired.
(As for those dependencies, they're submoduled in the repo: try `git clone --recursive`)
class DictObj(dict):
__getattr__= dict.__getitem__
__setattr__= dict.__setitem__
__delattr__= dict.__delitem__(Only one of our two statements is true.)
Similarly, you almost never see type checking; code typically just calls the method they expect to be there and hopes for the best. If you come from Java culture, this Is Not Cool, but it's Python, not Java. (I personally prefer Perl's roles for duck typing; I get type checking, but no implementation requirements.)
Anyway, I'm sure you worked hard on this, but it's one of those libraries that feels like it doesn't belong, like Twisted. You can ignore the Python community and be successful, but don't expect unlimited praise for doing so.
If we disagree, it's because I don't see "where built-in types will suffice" and immediately think "always", as perhaps you do.
This miscommunication is largely my fault. I shouldn't have marked the repository or its corresponding HN post with "Stop passing naked dictionaries around!” I must admit, my marketing impulse got the best of me there. Naked dictionaries with untyped values are obviously one of Python's strengths, and the very option of using them instead of a heavier system is a big part of why dynamic languages are so important, both to me and to the craft as a whole. It would be a more accurate reflection of my perspective if I had rather said "Stop always passing naked dictionaries around, especially in cases where you have to write schema validators for them!"
Statically typed languages are a drag because you've got type checking always, and you don't always need it -- in fact, as any dynamic programmer knows, you almost never need it.[1]
But sometimes you do. It's a fact that data schemas are useful at times. In instances where your data is in fact naturally schema’d,[2] I’ll invoke some PEP 20 dogma of my own: “Explicit is better than implicit.” There are situations in which you could catch a data problem at the time you’re storing it, but with dictionaries you only catch it later, once it’s being used (hopefully your test coverage surfaces it, or it might not even happen until you’ve deployed to production).[3] Recognizing this, you may then write a one-off schema validator for each of these situations, but that puts the onus on every developer to call that validator every time he or she is populating a dictionary of that classification (not very DRY).
In the six months or so since I wrote the first version, I’ve found DStruct to be extremely useful for lots of scenarios, like remote resources (I just wrote a use case here: https://gist.github.com/1398138), mock objects (I just wrote a use case here: https://gist.github.com/36118077f75eaed8c731), future-proofing, backwards-compatibility, and pretty much any time you’re doing outside-in development (e.g. writing a view or serializer first, and having it build up its own mock data objects, to be replaced by model instances once the model layer is complete).
In my heavily content-driven game framework, the game designer imports content via CSV.[4] With DStruct and a handful of code changes (amounting essentially to `return Decoration(input_dict)` or `return LevelMilestone(input_dict)`, instead of just `return input_dict`), we have been able to save enormous human time, because the server now detects schema-related content problems at content import time, rather than having to wait for the full content test suite to run (and hopefully have enough coverage that it can detect a missing or malformed attribute!).
I put it DStruct out there because it simply kept on saving me time, code, and technical risk in a variety of projects. Everyone is free to take it or leave it, but my preference is that they fully understand it before doing so.
-dorkitude
PS: I don't come from Java; I'm a Pythonist born and raised (even way back at IMSA, where I believe we met once or twice ;) I think in terms of metaprogramming, multiple inheritance, dynamic typing, and runtime mutation. I understand the kneejerk "oh god another Java bureaucrat" reaction, because it's a reaction I myself feel when I see certain patterns being hamfistedly jammed into Python code. DStruct is not such an occurrence.
[1] The debate about DStruct is not a debate dynamic vs. static typing in language design (we’re all pythonists here): it's a debate about the degree of the qualifier "almost" in this sentence.
[2] Sometimes you’re mocking something that’s schema’d; sometimes the interface is beyond your control; sometimes you’re retrofitting a class that used to be ORM and now is not; sometimes you’re importing CSV’s; sometimes you’re writing a feed aggregator; the list of naturally schema’d data sources goes on and on.
[3] This is particularly true in Ruby, where they always seem to use Hash-as-argument in an attempt to get the future-proofing power we get for free in Python via keyword args.
[4] And not just store items, settings, and game objects, but even the rules and economic structure! The game framework referenced here is the best work of my career to date. Unfortunately it is a closed source project, but suffice it to say the concepts baked into DStruct (and a handful of other killer libraries I will release in the coming year) had a significant positive effect in terms of code bloat, development time, quality, and risk.
If you want dicts to be accessible as objects with attribute access instead of subscript access, either create a real object which mocks/wraps/encapsulates/etc. your dict, or modify your desires.
This is not Pythonic. Sorry.
I have c# for when I want to pass around well dressed and proper things around…
There are benefits to using tuples over objects/dictionaries: 1) Lower memory consumption 2) Faster lookups 3) Can be used as dictionary keys, set members, anywhere a hashable (immutable) object is required.
https://github.com/pthatcher/pyrec
Here's my original post about it:
http://www.valuedlessons.com/2009/10/introducing-pyrec-cure-...
And here's my analysis of how much less memory it uses:
https://github.com/tantalor/megaera/blob/master/megaera/recu...
Can we now have a query mechanism for lists of these?
It seems to compete (unfavorably) with namedtuples as well.
pytyp has something similar - http://www.acooke.org/pytyp/pytyp.spec.record.html#module-py... - although that is just part of a wider framework for types. pytyp will also do things like use type annotations to map from json to python objects, for example, or check types on function args, or work with isinstance():
>>> isinstance([1,2,None,4], Sequence(Option(int)))
True
I released DStruct because it has been insanely useful in my work. Take it or leave it, but do so pragmatically: dogma is a terrible way to make decisions.
With that I must disagree :)
in other words, the library is the end result of a process of exploration of some ideas that interested me related to metadata and types in a dynamic language. it's not an attempt to make python like another language, although (of course) it borrows ideas from those languages (because people have already spent a lot of time thinking about how to make a type system orthogonal and elegant).
and because it's dynamic, metadata-based, uses reflection, and works with duck typing, pytyp's type checking is inefficient for global use. it makes more sense as a check when debugging than a global language feature.