*args and **kwargs in python explained | Hacker News Reader

*args and **kwargs in python explained

freepythontips.wordpress.com

63 pointsyasoob33 comments

I find it interesting to see all of the comments on HN about the dislike of args and kwargs, especially given their necessity for higher order programming.

If you are making functions that modify other functions (like decorators), these are highly useful. In Python you can't guarantee that all functions you will want to modify are going to have the same type signature, so you need to be able to gracefully handle all possible combinations and pass them on to your function body. Sometimes you'll just pass args and kwargs on to a wrapped function...other times you'll do things like pick out a single argument from kwargs and use it to do something useful. But make no mistake, in Python these come in handy to have!

Personally it feels like spaghetti code. Tornado for instance is a web server written in Python and there are classes that have many parents and kwargs is used everywhere to pass named arguments up to each parent class in the constructor. The problem is it's hard to get a grasp by looking at the constructor as to what arguments should actually be passed to a class to initialize it. There's no contract or spec enforced in python so it's easy to see how errors can crop up by missing a random argument a certain parent class is expecting. It also makes it difficult to understand what a class does when the arguments a function expects are not declared.

They are also easy to abuse and make APIs hard to figure out. You could see stuff like:

  process_payment_data(**kwargs)

In the docs and then have to go find out what possible values kwargs could take.

That is a bug in the docs, not in the use of kwargs. If the function expects certain keyword arguments it must document them. You can just as easily write specific keyword args, and then fail to document them in exactly the same way.

At least you can tell what the names of arguments are and if they have defaults or not.

The thing that I love about python is keyword arguments. It makes the code so much more readable. The wildcards, while powerful, reduce readability.

For me, a equally powerful AND readable way is to have named arguments with defaults. So when you change the function definition to add more arguments you can insert defaults so as not to break existing code.

I find the wildcards most useful when implementing decorators or other cross-cutting metaprogramming features. For example, you could easily do a tracing decorator like this:

  def trace(fn):
    def worker(*args):
      logging.info('Called %s with args %r', fn.__name__, args)
      return fn(*args)
    worker.__name__ = fn.__name__
    # other decorator boilerplate
    return worker

  @trace
  def foo(bar, baz):
    ...

  @trace
  def foo2(x1, x2, x3):
    ...

(Note that logging.info itself takes args & kwargs as its arguments, and passes them along as formatting arguments for the string.)

If you've ever looked at AspectJ or even Haskell, this sort of functionality is virtually impossible in a statically-typed language. You go through all sorts of contortions to be able to say "I want to operate on any type of function, and I don't care how many or what type of arguments it has", and then come out with something insanely complicated that satisfies nobody. For certain use-cases, you really don't care what type of arguments it takes, you just want to pass it through to some other function.

I do wonder if there's some other language feature that could be used to achieve this without breaking documentation or static analysis, though.

You can use the `functools.wraps` decorator to deal with copying all (most) metadata from the wrapped function:

    def trace(fn):
        @functools.wraps(fn)
        def worker(*args):
            logging.info('Called %s with args %r', fn.__name__, args)
            return fn(*args)
        return worker

Docs: http://docs.python.org/3/library/functools.html#functools.wr...

I've been looking for something exactly like this. It's so tedious trying to remember what metadata needs to be copied from a function when making a decorator. Thanks!

Also, BTW, I hate the practice of using default arguments to avoid breaking existing code, because it makes the code reflect how it got there and not what it does. You end up with a bunch of call sites, some that take n arguments and some that take n+1, and you have to look at the function definition to see what the n-arg call sites are doing. I would much rather use an automated refactoring tool to edit all call sites to take n+1 arguments with the default inline in the call, although this is a bit hard to do in Python.

In my experience, the best use for default arguments is when the API contract really does call for "default usage", but occasionally you need to pass an extra arg for special purposes. So if 95% of call sites only need n args but 1-2 of them need to vary a parameter or two of the algorithm, then defaults are appropriate. Passing the extra args by keyword usually ends up being much more readable in this case though.

You can also just give the n+1 argument function a different name, and implement the old version in terms of it.

I'm no expert, but I've found it's best to use kwargs near the middle of a class hierarchy. This makes it much simpler to build the internal API - the classes near the top of the hierarchy have specific kwargs, but their subclasses don't need to explicitly enumerate them. One can then write user-facing subclasses at the bottom of the hierarchy that have explicit arguments and keywords arguments.

I've always felt like the args and kwargs syntax in python is a blemish on an otherwise beautiful language. I'm wondering why not just let any function accept variable args, and reserve the keywords args and kwargs (without stars), where in a function body they refer to non-specified / explicitly defined arguments?

The only downside I see to that is the ability to write code poorly by passing junk arguments into functions that never get used. But you can do that with any function that declares kwargs or args right now.

Or just reserve args and kwargs and have it so functions with them in their arguments list have the same behavior as args an kwargs (just without the ugly stars). Maybe even denote them as __args__ and __kwargs__ in the same way you access special member functions of objects through __foo__ syntax, that would at least (to me) be consistent. I did a cursory google search if there is any language wide syntax surrounding star + variable name, and I couldn't find any, which seems to mean the name was just a hack using the syntax of a pointer dereference from C.

Am I wrong here and just missing some grand logic behind it? The rest of the language is really sensible so I probably am.

I don't know the history behind star args / kwargs, but I don't think what you're proposing is a good idea:

* You wouldn't get TypeErrors for broken code. Most code doesn't use or need star args / kwargs, but if I understand correctly your first proposal, it'd mean no TypeErrors on function signatures, making it harder to find coding errors.

* Python has very few keywords on purpose, and I don't think they'd like adding the fairly generic, non-keyword-seeming names like "args" and "kwargs".

* "Explicit is better than implicit" is part of the Python philosophy. The single and double stars may "look ugly", but that's part of the point. They're not just ordinary variables, and they do something special, so they need to stand out.

* Your proposal of using __args__ is at least explicit in their use, but not in the function definition.

For what it's worth, I mostly use star args / kwargs in wrapper or "proxy" functions: functions that do something like logging and then call through to another function to do the main work.

I work on Hy, a Lisp that compiles to Python AST, and my intuition tells me that it's a bit of a wart too. Particularly how the syntax for calling is conflated with the definition and partly in how the arguments are filled in the Call node (basically, left to right). It lets you do things like:

  >>> def foo(a, b="Default"):
  ...     print(a, b)
  ...
  >>> foo(1, 2)
  (1, 2)
  >>> foo(1)
  (1, 'Default')
  >>> foo(1, b="bar")
  (1, 'bar')
  >>> def baz(*args, **kwargs):
  ...     print(args, kwargs)
  ...
  >>> args = (1, 2, 3)
  >>> kwargs = {'foo': "bar"}
  >>> baz(*args, **kwargs)
  ((1, 2, 3), {'foo': 'bar'})

It's the first example that kind of seems 'off'. It's probably just cross-polination from my CL experience (which seems 'more correct'):

  ?> (defun foo (a &key (b "Default")) (list a b))
  FOO
  ?> (foo 1 2)
  Error: Incorrect keyword arguments

For the most part it hasn't been an issue when I'm writing the code. But when I read code that does this stuff on the call side frequently (or isn't consistent in its use over time) it can be a little frustrating.

Python 3 lets you define functions with keyword-only arguments:

    >>> def foo(a, *, b="Default"):
    ...     print((a, b))
    ...
    >>> foo(1, b="bar")
    (1, 'bar')
    >>> foo(1, 2)
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: foo() takes 1 positional argument but 2 were given

Right, I should've specified the Python version. Python 3's grammar for argument lists changed a bit and added kwonlyargs to the mix. IIRC the call node's arglist isn't just filled in left-to-right and handed names for keyword arguments anymore.

Python 3 is definitely a step in the right direction. I wish adoption was a little more rapid.

It seems like what you really want is for b to be a keyword-only argument. Python 3 provides this fwiw.

It's the same as self.

    Beautiful is better than ugly.
    Explicit is better than implicit.
    Simple is better than complex.
    Complex is better than complicated.
    etc.

There are a few implicit things in Python, but I would consider those the blemishes on the language.

If you need to check arguments passed into kwargs (and you do need kwargs for some reason) then why not, well, check them?

After a couple of years with Python I had a first use case for an ordered version of kwargs. I haven't really dived into the problem yet, but could this be done with a metaclass or is kwargs rooted deeper in the stack?

you can't do it. see the comments in the constructor for OrderedDict (if i am remembering correctly - somewhere there's something in the python source that basically says "rats, need an ordered kwargs but that's not possible").

here you go - http://svn.python.org/view/python/trunk/Lib/collections.py?v... - line 32.

probably you need to pass in an OrderedDict instance.

Yes, I thought so. Otherwise there would be much more ordered kwargs decorators or hacks floating around. Thanks for the link to the source.

Would named positianal arguments with defaults not work in your scenario?

I guess no. In my use case hundreds of keywords are possible and in the ideal scenario `f(a=1, b=2)` and `f(b=2, a=1)` could be distinguished and saved internally in the order they were given.

There are easy ways around this, e.g. just passing something ordered to `f`, but the keyword argument syntax seems the most concise to me.

Or maybe I do not get you approach - in that case, could you give an example?

Even if you figured out a way to do this, it's wrong wrong wrong. Everyone using an API has a natural and correct expectation that keyword arguments will be order-insensitive. Making them order-sensitive is just begging for trouble.

If they are, by default, order-insensitive, then it should pose no problem at all to make them order-sensitive, since you would not know what order they would appear in either way.

Now, if you had said the other way around (ppl expect them to be order-sensitive but they are order-insensitive), that would have made sense...

No. Because normal behaviour is order-insensitive, I expect that f(a=1,b=2) and f(b=2,a=1) should have identical behaviour. If you make the arguments order-sensitive, then they may not. If, for readability reasons, I change the order of the arguments, I will then be surprised when my code stops working.

Conversely, If (because I think it is order-sensitive) I believe that, in order to make the function do what I want, I have to call f(a=1,b=2) and not f(b=2,a=1), then I'm not going to change the order willy-nilly, so I won't be affected by my expectations not matching reality.

How about passing an ordered dict?

I found this article http://freepythontips.wordpress.com/2013/08/03/a-url-shorten... also interesting and well-written.

that has explained the mechanics of kwargs, but not explained why we would use kwargs. I hardly ever use kwargs because I think they make the source code hard to read.

I think they are useful for systems with big configurations (that grow with features during development). I give every customizable function a reference to the big master config dict. Still not sure if that's a good design or not ...

I've used a couple of times for high-order functions. For example, I was playing around with sound generation and decided to create a 'stream' object. Instances of this object were defined by a function that generates the sound wave. This meant that the constructor took a parameter a somewhat generic function. I also found that some of my generator functions could take arguments themselves, such as pitch. Having args and *kwargs allows you to pass these arbitrary variables through.

kwargs are useful when you don't know in advance what the argument names will be. for example, writing a proxy or decorator that is used on multiple signatures. or implementing some kind of trace facility where you want to log variable names and values.

Reply on news.ycombinator.com