Python decorators they don't tell you about | Hacker News Reader

Python decorators they don't tell you about

github.com

281 pointsemidln86 comments

This notebook should be named "how to abuse decorators". Honestly, there is very little here that you want to use in a real project.

I completely agree in that saying "a decorator is a function that takes a function and returns a function" isn't technically correct since you can do much more than just that; but for readability, maintainability and other real world purposes, you might want to keep to that definition.

I disagree. From the end section of "real-world usage examples", there is this library https://github.com/mrocklin/multipledispatch/ that implements "Decorators for registration. It's used in at least one (probably many more) serious open-source project that I know of: GPflow https://github.com/GPflow/GPflow/blob/master/gpflow/expectat... .

"Decorators for preconditions" look pretty neat as well. It's a little sad that the preconditions themselves have to be written inside strings though.

> It's a little sad that the preconditions themselves have to be written inside strings though.

Isn’t it possible to pass a lambda instead?

Yes. Lambdas in Python are awkward, but you can definitely use either a lambda or a named function.

In fact, you could define a number of useful precondition functions in a single module and use them throughout a project. A couple of higher-order functions could make the post's examples safer and nicer-looking too. For example:

    @precondition(starts_with('The year is '))

where the starts_with precondition is:

    def starts_with(param):
        def test(s):
            return s.startswith(param)
        return test

> Lambdas in Python are awkward

Can you elaborate on that, please?

Considering the examples given by the author, I think they would be a better option.

Not the OP but I think he was referring to the restrictions python puts on lambda expressions:

- they can only contain a single expression

- they cannot contain statements

For precondition checking, shouldn’t that suffice?

Depends on the precondition.

Python has lambdas that have multiple expressions and statements.

They're called functions.

Python functions aren't anonymous

I've used that multipledispatch, along my own homegrown version, in production systems.

"Decorators for preconditions" indeed looks neat, reminds me of Clojure which has both preconditions and postconditions built in.

See my comment above. And yes, the preconditions could be written with a lambda, which would be much cleaner than using eval.

Technically a better definition would be -- "A decorator is a callable object which takes in a callable object and returns a callable object" but I agree that it is much easier wrapping one's head around it when the callable object is a function and decorator just acts as syntactic sugar for higher order functions.

No, if you read the article, you can learn that a decorator is a callable which can return ANYTHING, not just callables.

Thanks, I stand corrected :) That makes me wonder , I have not been able to imagine a usecase for it when it doesn't return a callable, the article doesn't help in that regard either. Got some thinking to do I guess.

the article mention multiple examples, one of being the "registering things" when it returns the same function untouched, just put away the function instance in a list or something to use it later.

Returning a function would be returning a callable object, which is what I was alluding to. The section under "Decorator Toolbox" contains no suggested usecases for returning a non-callable object, will try and think of something.

The @route decorator in Flask has only the side effect of registering a route with the function as handler. I prefer to have a map for big projects where I can see with a glance what routes directs to what functions, but for getting started and for little projects I find it nice and reasonable.

I've used decorators for registration multiple times in the real world. It's nice to be able to visibly attach the registration function call to the object you are registering rather than having it just written after it somewhere.

> This notebook should be named "how to abuse decorators". Honestly, there is very little here that you want to use in a real project.

Annotation, registration, dispatch and verification are pretty much the most common uses for decorators. If using decorators for these is bad, then decorators aren't needed at all: function wrapping is much more rarely used than those other four, and would not warrant an additional syntax construct.

Annotation and registration are indeed common and proper uses of decorators.

What I dislike are mostly the inputs 8, 9, 13, 14, and 23 that could give nasty ideas to an inexperienced developer, while a more experienced developer wouldn't need them to understand that a decorator is just a syntactic sugar and that, yes, "applying a decorator d to a function x [is] the same as writing the definition of x, then x = d(x)" (which the author seems to refute).

I understand that those examples are solely intended to show that "it works", but they aren't educational to me.

I disagree. Actually I am not a big fan of decorators since for me they tend to obscure WTF is going on in other people's code I'm reading. However, I quite liked the section "Decorators for verification" -- I hadn't seen that use case before and it seems handy to me.

If you're on mobile, view the page as a desktop site in your browser to see the IPython notebook rendered properly. This works on iOS at least.

Same in Chrome on Android. Thanks!

Consider functools.wraps[1] in your own decorators. This uses functools.update_wrapper to copy the name, docstring, and some other other attributes[2] from the wrapped function to the wrapper.

[1]: https://docs.python.org/3/library/functools.html#functools.w...

[2]: https://docs.python.org/3/library/functools.html#functools.u...

EDIT: s/wrapped/wraps/; tip of the hat to @ramnes

Yes! It's functools.wraps*, though. :)

URL for mobile: https://nbviewer.jupyter.org/github/hchasestevens/hchasestev...

I've used decorators mainly to retry a function if it throws an exception, like retrying DNS queries multiple times, etc. I've never got a chance to do more with them, but this article gives me an incentive to try more.

That never crossed my mind. Neat trick!

Yep. I do the exact thing for a lot of network related things (dns lookups, http requests). I use this nifty library: https://github.com/jd/tenacity

Wow, both terrific tips, thanks!

My favourite library for writing decorators that "behave nicely" is wrapt – https://wrapt.readthedocs.io/en/latest/

There's also a wealth of resource in the docs!

Medium to large Django projects were the first times that I began to see the power and usefulness of decorators. The most useful one I've used was a decorator to require certain JSON parameters in a POST and optionally specify a list of possible parameters. Makes the API code logic infinitely simpler, especially if you write some custom middleware to handle / report API errors. The code is clear and declarative, and errors are handled gracefully all in one place.

Which is very funny, because you'd get the same without decorators by simply using properly defined RPC protocol instead of ad hoc developed entangled mess that passes as REST.

You still have to define the procedures to use with the protocol using language features - like decorators. For example: http://spyne.io/docs/2.10/reference/decorator.html

functools.lru_cache[1] from the standard library is a decorator that memoizes[2] a function.

https://github.com/osteele/callgraph/blob/master/examples/ca... has some examples of using @lru_cache, together with a decorator (@callgraph) defined in that repo, that helps visualize recursion and memoization.

[1]: https://docs.python.org/3/library/functools.html#functools.l...

[2]: https://en.wikipedia.org/wiki/Memoization

WTH is that call to os.system("rm -rf /") doing there?! Issued a PR.

Reading the text around it makes it pretty clear: it's code that shouldn't ever be called.

It’s still widely inappropriate.

It is showing confidence that the author know what he is talking about, because as you can see he ran the code and that part did not run.

jupyter notebooks are easy to share and run locally, so it's still inappropriate

curl | bash is easy to run locally too, and very few people actually read the code that runs to check for “rm -rf /“-type things. When executing untrusted code from the internet, either you read it or your fate is in the author’s hands. No exceptions, not even for beginners getting started in programming via jupyter notebooks.

I agree with all the above, but I am more concerned with what's given out. Not a great idea for a tutorial IMO. You can make the point without having to actually using that command - just make it clear as a comment. If this were some other doomsday commands, you are screwed too. Just be nice to your readers.

People also run compiled code all day long. Curl | bash gets a lot of derision for being a bad practice. Unlike other applications that install themselves the one advantage of curling to bash is that its trivial to read the code.

Well, then let's go with the fact that a function called "never_called" is trying to delete all the files on the computer. At best it's a misnomer.

And it is right that the fate is in the author's hands. It's those hands that we are trying to tame.

if you run that code, it still won't run :D

All I get is that it's code that is not ever called, not that it shouldn't be.

Yeah I was going to issue a pull request also, it requires --no-preserve-root on most Linux systems to work as expected.

> Issued a PR.

Hmm? I don't see any such PR.

Here: https://github.com/hchasestevens/hchasestevens.github.io/pul...

> > Issued a PR.

> Hmm? I don't see any such PR.

Hmm, neither do I.

I've been exposed to decorators (starting with Java) and have 'gotten' them and I will proudly admit that I despise decorators and don't think they are great at all.

Just hate them? Did they steal yoUr lunch money? Or is there something more specific you hate about them?

Mate, you've got some broken images on your homepage.

Thank you for the heads up! Will fix when I get home

Like many language features, they can be extremely useful and can significantly improve code clarity.

They can also be abused in such ways that clarity is actually destroyed.

It's all very well writing fancy decorators, but unittesting them, specifically, mocking decorators is trickier.

I had fun a little while back abusing decorators to alter the return value of generator functions: https://sites.google.com/site/bbayles/index/decorator_factor...

Uploaded to Azure notebooks in case anyone wants to run/play with the code:

https://notebooks.azure.com/smortaz/libraries/PythonDecorato...

Click Clone & Run.

one guy made a "pure" decorator that used the ast module to assess the purity (to a certain extent) of a function and allow optimisation on the way

fun

crazy and fun

I once wrote a decorator that translated the function to Javascript and spawned a web server, allowing the function to be computed in parallel by multiple browsers: https://bitbucket.org/robertodealmeida/parallejax

cute one

Decorators are great. One or two or three. 4 is pushing it. Some people get all carried away though. I saw code recently that had a stack of like 8-10 decorators on top of each function. More decorators than function in every case. Thinking it might be time to re-evaluate basic structure in this case but maybe I just don't get it.

> Nothing in x = d(x) necessitates that d is a function - d just has to be callable!

Python being duck-typed, what's the distinction?

The distinction is that ‘function’ is a type which you can check as in

    isinstance(obj, types.FunctionType)

To point out that other callable things can be used. You would not say that "classes are functions" I think, yet they are callable.

"callable" essentially just means you can put a "()" after the name in the code and it works.

You can "call" a class. Doing so allocates an instance of that class (Or calls the __new__ function if one is defined) and returns it, first calling the __init__ function of that class if necessary.

You can also "call" an object if its class has a __call__ function defined.

Indeed, but isn't that exactly what "quacking like a function" is?

The funny thing is that all of the "useful examples" were functions that took functions and returned functions.

Can you make a decorator that references a class instance and has parameters?

Yes on the former (works just like you'd expect); to make a decorator take parameters, to do something like:

    @decorator_with_params("Some string")
    def my_function():
        pass

you have to make 'decorator_with_params' a function that returns a decorator; i.e.

    def decorator_with_params(param):
        def real_decorator(function):
            def inner(*args, **kwargs):
                print(param)
                function(*args, **kwargs)
            return inner
        return real_decorator

(You can also do something similar with callable objects, but that's ickier IMO.)

Demo of both (decorator as instance method and decorator with parameters): https://gist.github.com/jonathonw/dac88b715aca4e4f1b2f513e1f...

So just a function that returns a closure? It can be a real pain to read code like this if it's overdone -- I really would need to trust somebody to let them go meta at their discretion.

The key is to have powerful ones that you re-use a lot. Then most of the time you just have simpler functions with standardized annotations.

I can't see any reason you can't do something like:

    class MetaDec(args, here):
        ...
        def __call__(self, func):
            # Decorate function here

    def meta_dec(args, here):
        return MetaDec(args, here)

    @meta_dec
    def decorated:
        ...

(apologies for any syntax fails, python is not my best language)

At which point one would imagine you can abstract out the __call__ part and just have a 'decorate' routine on any class designed to work what way or similar.

Whether this is an improvement or not over closures almost certainly depends on the use case, of course.

Does the code actually work? Wouldn't you need to create a instance within meta_dec and return the result of the __call__ method. See at first glance I won't think it would work if I didn't know __call__ is a magic method in python. I don't think I'm being overly picky but I found the previous example much easier to read -- since it's a pattern I defaulted to myself before across several different languages. Mixing classes and FP like that doesn't resonate well with me tbh, but yeah you could do it ... but should you?

> Does the code actually work?

Given its reliance on '...' and the fact I'm not great at python and typed it straight into the comment ... no. No it doesn't.

> Wouldn't you need to create a instance within meta_dec and return the result of the __call__ method.

No, like with ES6 parameterised decorators return a function that then gets called with the thing being decorated.

> See at first glance I won't think it would work if I didn't know __call__ is a magic method in python.

(1) my reply was assuming the context of us both having just read the article, which has an example with __call__

(2) as I said in my original comment, you would abstract this away (and document it), I was sketching out an alternative implementation path to allow you to have an object instead of a closure since you didn't seem to be happy with closures, not offering a complete solution

> Mixing classes and FP like that doesn't resonate well with me tbh, but yeah you could do it ... but should you?

(3) if you don't like closures and you don't like objects, I'm not sure what other combined function+data abstraction is available in python that you wouldn't also dislike

(4) I'd probably consider doing it if I had a bunch of similar decorators and found the functional approach was starting to groan at the seams - I use objects for 'parameterised bag of functions, possibly with inheritance' quite a lot for that purpose

Sorry, must of given off the wrong impression. I'm quite happy with FP in general, I'm just aware it is possible to write some pretty horrendous things. But yeah you are completely right about (4), at some point it doesn't seem like overkill anymore.

I was very much trying to find an answer to your

> It can be a real pain to read code like this if it's overdone

wherein (4) is, when things start getting complicated, a way to end up avoiding things turning into closure soup. Also being able to selectively override things with inheritance can really help factor things out and clean things up.

> at some point it doesn't seem like overkill anymore.

Though I suspect "at that point, ask yourself if you're overengineering it and only if you're sure you aren't consider moving to a class" definitely applies :)

You mean something like:

  class FancyDecorator():
      __init__(self, *args):
          self.args = args
  
      __call__(self, fn):
          return decorate_fn_with_args(fn, self.args)
  
  dec = FancyDecorator(42)
  
  @dec
  def foo():
      pass
  
  @FancyDecorator(41)
  def bar()
      pass

Might have it wrong since I haven't messed with decorators in a while and they aren't exactly what one would call simple.

--edit--

code formatting is all wonky so...

http://chimera.labs.oreilly.com/books/1230000000393/ch09.htm...

Python decorators are like poor man's method combination.

They're a kind of advice [1]. Advice was indeed subsumed by CLOS method combination, but existed prior to that and has a life of its own, for example in Aspect Oriented Programming.

[1]: https://en.wikipedia.org/wiki/Advice_(programming)

[2]: https://en.wikipedia.org/wiki/Aspect-oriented_programming

To be honest, decorators were always, idk, I felt like there was something off about them.

> To be honest, decorators were always, idk, I felt like there was something off about them.

I think they're one of those things you don't need to use unless you need to use them...though, honestly, you probably don't ever need to use them unlike metaclasses where sometimes they are the only way to do something in a reasonable manner.

> you probably don't ever need to use them

By their definition, you don't ever need to use them. They're just a convenient syntactic sugar.

  @foo
  def bar():
      ...

is, as far as I know, exactly equivalent to

  def bar():
      ...
  bar = foo(bar)

Reply on news.ycombinator.com