Python: The Dictionary Playbook | Hacker News Reader

Python: The Dictionary Playbook

blog.amir.rachum.com

193 pointsNurdok55 comments

You mentioned it a bit, but I want to make it clear that, even if you don't use 2.7, you can still count by doing:

    from collections import defaultdict
    counter = defaultdict(int)

There is a difference though, because you have to count manually, i.e:

    for i in 'supercalifragilisticexpialidocius':
        counter[i] += 1

Also, because defaultdict accepts any callable, you can have a dict of counters by doing:

    counters = defaultdict(lambda: defaultdict(int))
    for word in ['apple', 'berry', 'grape']:
        for letter in word:
            counters[word][letter] += 1

This is not very obvious, so I don't use it a lot, but sometimes it's the most elegant solution.

There's also an implementation of Counter for 2.5+ on ActiveState: http://code.activestate.com/recipes/576611-counter-class/

For Python 2.7 there's collections.Counter [1]:

    import collections
    c = collections.Counter('supercalifragilisticexpialidocius')
    print c
    # Counter({'i': 7, 'a': 3, 'c': 3, 'l': 3, 's': 3, 'e': 2,
    #          'p': 2, 'r': 2, 'u': 2, 'd': 1, 'g': 1, 'f': 1,
    #          'o': 1, 't': 1, 'x': 1})

[1] http://docs.python.org/2.7/library/collections.html#collecti...

It's odd that with all the 'high-brow' datatype names such as dictionaries, tuples, and sets, they decided to call this one Counter instead of the obvious choice 'multiset' (or 'bag'); it supports set operations such as intersection, so it's more generally useful than just collecting top ten lists &c.

You know a guide is good when it makes you want to go back and refactor old code. Great information, thanks :)

Interestingly after reading the post I've realized that while I've been using the more advanced stuff, I was missing the first 2 basic ones.

Happened to me too, I always used 'for key in mydict: do_something(key, mydict[key])' instead of 'for key, value'

I guess python leaves so many ways to do things that you tend to settle on some style quickly even if it's not the most efficient one.

When self learning new languages I still miss an efficient way to get all the idioms. However I think that in python it's not really that important, as long as you get stuff done - your code is still going to be quite readable.

Python doesn't always hew too closely to TSBOAPOOOWTDI, despite Ken Peters' lovely entreaty. http://www.python.org/dev/peps/pep-0020/

My good friend, refactoring is sometimes good, but remember the ancient wisdom: don't fix what's not broken! Your old code may not be pythonic, but still it works...

This is a really nice guide. I've been up to my eyes in python dicts over the last few days so already had most of these figured out.

setdefault is new to me, which is cool. Unfortunately I can only see one place to use it in my code and it would be inefficient [0]. Best stash it away for later use :)

[0] r = re_subs.setdefault(s, re.compile(s))

iirc re caches regexps. so unless you're using many (more than the cache size [edit: 100]), i wouldn't worry (well, and also measuring before optimising...)

I did wonder if that was the case but I couldn't find anything about it. Just discovered re._MAXCACHE which implies it does cache. Tested and in my case the performance is the same.

Thanks for the tip (I was posting half hoping someone would have a better solution). That's another 3 lines of code removed :)

Good advice. I've got one to add:

  #x and y are dictionaries
  z = dict(x.items() + y.items())

It merges two dictionaries, giving precedence to the second (in Python 2 - Python 3 is a bit more nasty: http://stackoverflow.com/questions/38987/how-can-i-merge-uni...).

Yeah, I like:

    z=dict(x)
    z.update(y)

It's clear and concise, and it's obvious which one gets precedence, even if it's two lines.

I tend to be partial towards

    z = dict(x, **y)

if the keys of y are compatible with unpacking.

Love defaultdict. It and dict/set/list compressions are a big part of what makes Python so fast to write in.

Great practice for 2.7 that's probably quashed in 3.0. For large dicts, no need to create a giant set en route when iterating over keys, values, or both. Use "for k in d.iterkeys()", "for v in d.itervals", "for k,v in d.iteritems."

While I'm at it-- if you're ever finding yourself using a huge amount of awfully rigid objects from a single class, use __slots__ to allocate needed variables! Python will otherwise define the object's namespace in a dict (called __dict__) which allocates a whole kilobyte per object. Bad news if you have several hundred thousand... Guessing this is why Guido loves namedtuples so much for basic attributed storage.

In Python 3.3:

Iterate over keys: "for k in d.keys()" ...over values: "for v in d.values()" ...over both: "for k, v in d.items()"

Love it. It's that way with range, too, right? Does one end up casting things to lists more often if so much is naturally an iterator?

http://www.python.org/dev/peps/pep-3106/

  The original plan was to simply let .keys(), .values() and
  .items() return an iterator, i.e. exactly what iterkeys(),
  itervalues() and iteritems() return in Python 2.x.
  However, the Java Collections Framework [1] suggests that
  a better solution is possible: the methods return objects
  with set behavior (for .keys() and .items()) or multiset
  (== bag) behavior (for .values()) that do not contain
  copies of the keys, values or items, but rather reference
  the underlying dict and pull their values out of the dict
  as needed.

"for k in d" is even shorter. I wonder what they point of the .keys() method is actually, perhaps it's just as redundant as .has_key(). Most times you can iterate over the dictionary itself; when you need to explicitly pass an iterator or list, iter(d) or list(d) is shorter than d.keys().

If you iterate over .keys(), you are allowed to use del key[value].

Only in Python 2, which returns something to the effect of list(dict); so my point remains, they keys method doesn't add anything which you don't get (more explicitly) by coercing to a list when needed.

I love curiosity sparking posts like this.

>>key in dct

is much better than

key in dct.keys()

Of course, that got me curios to find out if there is a magic method out there that takes advantage of keyword "in". Turns out __contains__ does that.

Always exciting to stumble upon new stuff in my favorite language.

I hate to be that guy but what about dictionary comprehensions?

Or dictionary views: they stay current as the dictionary is updated, and allow set operations.

Have used all of the techniques in the article (plus dict/set comprehensions) but had never heard of dict views [1] until now. Good stuff!

[1] http://docs.python.org/2/library/stdtypes.html#dictionary-vi...

I don't get it. Why would I use a view instead of the dictionary itself?

http://stackoverflow.com/questions/340850/python-3-0-dict-me...

Views are lighter than a full copy of a list, yet behaves like a list (eg: supports `key in view`).

[edit] Also, this seems to be the relevant PEP:

http://www.python.org/dev/peps/pep-3106/

It's not meant as a replacement; views form the new implementation of the keys(), values(), and items() methods in Python 3. The have been made available in Python 2 as viewkeys() etc. As said, they have the advantage of providing set operations (e.g., intersection with the keys of another dictionary), and they are more efficient than keys() or iterkeys(), because they have less overhead.

One my favorites, safe deep searching: by returning a dict you can run another .get

var = {'a' : 'b' , 'c' : {'d' : 'f'}}

print var.get('c', {}).get('d') print var.get('DNE', {}).get('d')

"not key in dct", or "not (key in dct)", is never slower than "key not in dct" because when key is found in dct, the expression is immediately true.

Perhaps, but it reads oddly which is reason enough to avoid it.

They are equivalent, but the latter version is more readable.

Those actually generate the same bytecode in Python 2.7. Probably other versions, as well.

In item 3, the "boilerplate" and the "awesome way" are not equivalent. The boilerplate does the equivalent of setdefault, which is mentioned later

    dct[key] = dct.setdefault(key, 0) + 1

Since you're assigning to dct[key] anyway, it doesn't really matter.

In most circumstances for (key, val) in data won't work, as the default dictionary iterable only contains the keys. You want for key, val in data.items():

Or iteritems() for an iterable.

I use dictionary comprehensions personally but I have mixed feelings about the syntax. It looks too much like set comprehensions on first glance. Compare the following to see what I mean.

myset = {x for x in "This is my stuff".split()}

mydict = {x:len(x) for x in "This is my stuff".split()}

Well I see what you mean but then again a comma looks very much like a period ...

The advantage of this syntax is that : unambiguously introduces a key: value pair, whereas (key, value) could also occur in a list comprehension (e.g., by accident).

Erm, you can leave out 'in most circumstances' and 'default'; this the way they always work.

well,

    (key, val) in data

works if

    data = {(1,2) : 3}

Well but then the variable names are just wrong.

Nice! Some good information here. Good for beginners, good for intermediates, some expert will probably say it's good for them too.

Thanks for writing this; it's very useful.

Are you planning on doing similar posts about other parts of Python in the future?

Can logical operator in Python be used to conditionally initialize an item in a dict? For example, can:

    group = dct.setdefault(key, []) 
    group.append(value)

be replaced with some equivalent of this Ruby snippet:

   (dict[key] ||= []) << value

?

No.

But of course if ``group`` is not used afterwards it can be inlined to

    dct.setdefault(key, []).append(value)

Thanks, I didn't know about "x not in D"! On the same note:

http://dacavtricks.wordpress.com/2011/05/23/python-default-v...

I also find the combination of lambda and defaultdict quite useful:

  d = defaultdict(lambda: False)

  d = defaultdict(lambda: {'foo':set(), 'bar':False})
  d['baz']['foo'].add(1)

For the first I do defaultdict(bool) ;-)

Pretty cool, bookmarked. I always enjoy different forms of presenting the info, playbook was a nice touch :)

my Bitdefender is blocking the site; says it's insecure.

nice! very useful, thanks for putting this together.

This is ... wait for it.. Legendary. :)

Rocking it out! Awesome! Getting down! Make it happen!

Reply on news.ycombinator.com