Hacker News Reader

Comment by emef | Hacker News Reader

I'm surprised using mutable value for a default in a keyword argument wasn't mentioned. This certainly tripped me up early in learning python:

    def fn(x, my_dict={}):
        my_dict[x] = x * 2
        return my_dict

    >>> fn(1)
    {1: 2}
    >>> fn(2)
    {1: 2, 2: 4}

(I would have expected the second call to return {2: 4} when I was learning python)

View on HN

This was discussed pretty recently in this thread on "Python Newbie Mistakes, Part 1".

Default values for functions in Python are instantiated when the function is defined, not when it’s called.

Thread: https://news.ycombinator.com/item?id=5999772

Direct link: http://blog.amir.rachum.com/post/54770419679/python-common-n...

It looks like the author posted a "part 2" to HN as well, but it never made the front page.

http://blog.amir.rachum.com/post/55024295793/python-common-n...

This is pretty bizarre, and seems like it would cause a memory leak if you don't know what you're actually doing. You're allocating a my_dict object on the heap, in a field of the function object, so my_dict never goes out of scope and never gets garbage collected until the function fn itself does, right?

Whereas if you did this instead:

    def fn(x):
        my_dict={}
        my_dict[x] = x * 2
        return my_dict

You'd get what you'd expect--a new my_dict object gets created, returned, and then goes out of scope every time fn is called, so it would get garbage collected once there are no more references to that return value. (I think...)

(I don't know that much about how memory allocation and GC works in Python yet, just trying to learn!)

Basically. The idiomatic solution for this in python is:

  def fn(x, my_dict=None):
     if my_dict is None:
         my_dict = {}
      ...

You can use a sentinel if you want to allow None as an argument. More on that, along with a way to use mutable defaults productively:

http://effbot.org/zone/default-values.htm

this is not at all bizarre, you just need to know what dynamic means. the best answer i could come up with is "definition is execution". for an enlightening moment, see this piece of code:

    def a():
        print "a called"
        return []

    def fn(x=a()):
        x.append(1)
        print x

    fn()
    fn()
    fn()

i suggest typing this directly into the interpreter instead of a script for better effect.

Great explanation!

You got me thinking about how python differs from lisp in this respect, and the subtle way that a functional API has helped this specific situation.

While lisp suffers from the same "definition is execution" gotcha, the effects are far rarer in practice because () is immutable and interned while [] is mutable and usually generated afresh each time it's executed.

  $ python
  >>> [] is []
  False

  $ sbcl
  * (eq () ())
  t

Since [] is mutable, appends of [] can do superficially 'the right thing'.

  >>> a = []
  >>> a.append(4)
  >>> a
  [4]

  * (setq a ())
  * (nconc a '(34))
  (34)
  * a
  ()

But there's a reason lisp does seemingly the wrong thing. According to the spec, nconc skips empty arguments (http://www.lispworks.com/documentation/HyperSpec/Body/f_ncon...) Reading between the lines, I'm assuming this makes sense from a perspective in lisp where we communicate even with 'destructive' operations through their return value. This is more apparent when you consider nreverse:

  * (setq a '(1 2 3 4))
  * (nreverse a)
  (4 3 2 1)
  * a
  (1)

Destructive operations can reuse their input, but they're not required to maintain bindings. There is no precise equivalent of python's .reverse(). Instead, a common idiom is:

  * (setq a (nreverse a))

It seems like a weird design decision, but one upshot of it besides encouraging a more functional style is that this optional-arg gotcha loses a lot of its power in lisps. It's very rare to define a default param of a non-empty list, and empty lists can't be modified without assigning to them.

I never did it myself, but this behavior is very useful if you want something to be used across invocations, like a cache.

To limit its size, you can use the ideas on http://stackoverflow.com/questions/2437617/limiting-the-size...

In other words, defaults are static (an in C) and initialized when the function is defined, not when it's called.

It is perhaps more correct to say "when the function is instantiated". Functions in python aren't exactly "defined". Functions are objects same as everything else. The "def" keyword is how you call the constructor for a function object. The kwarg default values are evaluated once at object construction time and saved.

The "def" statement may construct different distinct function objects from a single definition. Each distinct function object has different default value objects, but each function object reuses its own default value objects each time it's called.

Consider this:

    def outer_fn():
        def inner_fn(foo={}):
            # The id() function returns an internal
            # object identifier.  Different objects 
            # have different ids.
            print id(foo)
        return inner_fn
    
    inner_fn_1 = outer_fn()
    inner_fn_2 = outer_fn()
    inner_fn_1()
    inner_fn_1()
    inner_fn_2()
    inner_fn_2()

You are absolutely correct, of course. I was using a C-ism.

So it's instantiating the dict in the declaration and recycling it between invocations? I wouldn't have expected that either.

Rather the dict is bound during "compile" time, and not at each individual invocation (which would be much better, IMO). And during each invocation, it doesn't get re-bound.

it can't be, that wouldn't make sense. "def" is no different than "for" or "if".

"def" in Python is an executable statement, not a declaration- it creates a function object that gets bound to the specified name in the local namespace. The defaults are evaluated when the function is created.

It's useful for memoization though and is where I've typically used it.

Reply on news.ycombinator.com