A primer on Python decorators | Hacker News Reader

A primer on Python decorators

thumbtack.com

187 pointsmidas41 comments

There is a small mistake in the post: decorators cannot be arbitrary expressions. Something like @Foo(spam).bar fails with an unhelpful SyntaxError, something that everyone who designs complex APIs will probably encounter at some point. This is a restriction which is in place because Guido didn't like arbitrary expressions as decorators.

Oh, Guido. Thanks, I've updated the post.

Thanks, but my favorite explanation of decorators is still this stackoverflow answer (see the second answer)

http://stackoverflow.com/questions/739654/understanding-pyth...

Yes, I think this post is more informative.

Direct link: http://stackoverflow.com/a/1594484/378825

What I really want to know is how a framework such as Flask uses a decorator for the route. How is the correct function picked for a particular route that is defined against the decorator? (Maybe I'm completely misunderstanding this...)

The decorator can do anything. In flask the decorator syntax is just shorthand for adding the route to a registry which is looked up every request.

The source code is pitifully small: https://bitbucket.org/mitsuhiko/flask/src/4d82231621fc/flask...

Calling a decorator can mutate some shared state.

Here's the source of Flask's `route` decorator:

https://github.com/mitsuhiko/flask/blob/master/flask/app.py#...

When you call `route` with a URL pattern, it returns an inner function which is used as the decorator. That decorator just records your route and function in the Flask URL map, and returns your function unchanged.

So, Flask is arguably perpetuating a slight abuse of decorators, since it doesn't decorate or wrap your function at all, but merely saves a reference to it somewhere. But it's a fairly clean way to make up for the lack of code blocks or multi-statement anonymous functions in Python.

Some might say it's a dangerous abuse of decorators.

Since decorators (generally) run at module load time, any stateful decorator (usually) implies the use of global mutable state, which is (considered by many to be) the root cause of much bad design, convoluted flow, limited reusability and untestability. This is perhaps why most decorators in the standard library are pure (off the top of my head).

This is well beyond the scope of the post but an important and often overlooked point in my opinion.

In flask, the app.route decorator mutates the app object. There are no globals necessary. Use of globals to maintain state is an orthogonal issue to use of decorators to update registries.

Well either the app object is global, in which case you've got global mutable state, or you're defining your handler functions later than module load time, which is pretty uncommon practice.

You can use the return value of some function as a decorator, as a way to avoid global state and tie the decorator to a given instance of your routing object. (I don't know flask, but this isn't a limitation of python)

While you're right it is an easy to make mistake with decorators, I don't think the issue is global mutable state as much as uncontrolled mutable state, especially if used against singleton.

Typically, doing this:

    @register
    def foo():
    ….

is bad, but this is much better:

    @registry.register
    def foo():
    …

if registry is a global object that is not a singleton. In that case, you can easily use distinct registries (for testing, etc…) and this is not much of an issue in practice. Another way of doing this kind of things is to post_pone the registration, with the decorator just labelling things:

    @register
    def foo():
    …

    def register(f):
        return WrappedFunc(f)

and then build an explicit list of modules being imported, and look for every instance of WrappedFunc in that module locals(). I use this in my own packaging project where people can define multiple python scripts that hook into different stages of the packaging.

For some reason, decorators didn't click for me until now. I've been parroting the standard decorator logic (e.g. returning a function) without knowing why.

Thanks!

To authors: I would avoid try except in this code snippet, a simple if else is more explicit. I would also avoid a = b = c statement. One line per statement is better most of the time.

The Python community generally advocates an "it's easier to ask for forgiveness than permission" coding style. When faced with a condition of the form "if condition a holds, do b, else c", it's very often a better idea to do "let's try b, and do c in case b fails because condition a didn't hold".

In this case it's better because you can avoid computing an extra hash of the object in cases where it's already a key of the dictionary. This may seem like a silly optimization, but it can very easily add up if you're accessing existing elements most of the time--I once had a bit of code that went 60x faster when I replaced an if-else with a try-except.

In other cases it can be even more beneficial. Say you're opening a file. One approach to avoid errors would be to check if a file exists first. This is error-prone because the file might cease to exist in between the 'if' and the 'open' statements, and now you have no code written to handle the error. Using try-except will ensure that you actually handle the error intelligently.

This isn't to say there's never a good reason to use an 'if' to check things, just that if you can do it in one step instead of two, one is usually better.

Agree with that "try except" is better than "if else" for file handling ("with" is even better).

I am not sure if "try except" is really faster than "if else" in some edge cases in a memoization context, as you claim.

What I am sure is that in a didactic context, where you want people to understand code, something like:

    if args not in stored_results:
        stored_results[args] = fn(*args)

    return stored_results[args]

is much better than that (from the OP):

    try:
        # try to get the cached result
        return stored_results[args]
    except KeyError:
        # nothing was cached for those args. let's fix that.
        result = stored_results[args] = fn(*args)
        return result

For a memoize function, try/except is perfect. You gain a slight speed benefit from function calls that are memoized as compared to the "if x not in y", and since memoize is all about speeding up repetitive function calls, it's the most optimal choice. Using the "if x not in y" you are giving up speed of the "x is actually in the y" in exchange for a cheaper than "except", however if you are memoizing a function, chances are that the function is going to eat up processing power making your optimization of the "exceptional case" at the expense of the "normal case" not worth it.

As far as people understanding code - this is a common pattern in python code, try / except should be easy for anyone to understand.

Out of curiosity, in a situation where you are doing negligible condition testing, is try–except still considered "more performant" than if–else flow control? For some reason (and I'm going to do so reading to get clarity on this point), I have had the silly notion that throwing and handling exceptions can be costly.

I don't think the try-except is 'more performant'. At least from the below benchmark test it doesn't seem to be so.

  >>> from timeit import timeit
  >>> timeit(setup='x=dict([(i,i*2) for i in range(10)])',stmt=
       """
          if 20 in x:
             pass""")

  0.07420943164572691

  >>> timeit(setup='x=dict([(i,i*2) for i in range(10)])',
       stmt="""
               try:
		  x[20]
               except KeyError:
		  pass""")

  1.1514457843105674

I am on my phone and cannot test this, but the try:except: construct is optimised for the non-exceptional path. The latter is probably faster for x[0] than x[20].

I'm no pythonist, but I think setdefault would be perfect for the case

http://docs.python.org/library/stdtypes.html#dict.setdefault

I agree. The author uses exception for something that is not at all exceptional. In fact, it will occur at least once. This is misuse of an exception. No one expects Spanish Exception!

Earlier, he uses the map function for no apparent reason. I mean - list comprehension would perfectly fit here.

I still think I have something to learn from the article, though.

The try/except is the preferred python way of doing things. In Python, try/except is cheap, and in the case where the try actually succeeds, you can gain a slight performance benefit. In this case, since performance is the goal, it is perfect for memoize.

I think subtle optimization should be left aside in a tutorial. Try except is a good way to handle many cases that are not exceptions, like when reaching the end of an iteration, but I never heard that it was "the preferred way" for checking the presence of a key in a dict. I believe the "if in" construct is cleaner, clearer and "pythoniker", if you don't mind the Housism.

"if in" is not the preferred way to check for the presence of a key in a dict. You should use setdefault. The thing is, you aren't searching for a key in a dict only. You're memoizing. And try/except is the most efficient way to do that. This article isn't teaching you how to program, it's teaching decorators.

Nice description. I'd suggest explaining how decorators that accept arguments (i.e. @memcached('some-arg')) work lest it befuddle some beginner. It is not straightforward (the first argument of a decorator is the function being decorated).

For those who prefer video, Dave Brondsema gave an excellent talk on decorators and custom context managers at PyCon: http://pyvideo.org/video/883/decorators-and-context-managers.

Nice description of method decorators. Didn't touch on class decorators, or decorators that can decorate both classes and methods (arguably very ugly wrapper functions that return decorators that decorate).

Wrote a post a while ago that talks about them - http://hangar.runway7.net/decorators-wrappers-python

For an interesting use of Python decorators - event handling - have a look at Decovent on pypi http://pypi.python.org/pypi/Decovent.

This is similar to how Pyglet does things: http://pyglet.org/doc/programming_guide/hello_world.html

Great primer, reading it gives a good sense for why the decorator syntax was made and the potential uses. Is this part of a wider series you are running?

Thanks! It is indeed the first in a series of Python posts that will be on the Thumbtack engineering blog.

Can you please email all future posts to me? ;p zackster@gmail.com -- I don't use RSS any more, and I'd hate to miss them.

Or just let me use Feedburner to subscribe via email.

Love using Python, but still haven't got around to using decorators yet. I probably didn't understand the concept properly until now ...

You could say decorators are for the code that you want to put inside a function but that don't really belong to its logic. The memoization example shows that: if you have memoization logic inside the function it works but you should feel that there are two disjoint logics at works.

Minor nitpicking:

> unlike in Java, you can also call a class method on an instance

It's possible in Java, too. It's just considered bad practice.

Still, a very nice read!

I'd say if you have to do that, it is most likely because you have a design problem somewhere. It is a code smell in Python as well.

Reply on news.ycombinator.com