Hacker News Reader

Comment by softwaredoug | Hacker News Reader

I'm curious if anyones found a good solution to Python default args?

For example, this doesn't do what people expect:

    def foo(bar=Bar())
        ...

Because this creates exactly one Bar at function definition time, not a Bar per function call.

Instead what people usually want is:

    def foo(bar=None)
        if bar is None:
            bar = Bar()
        ...

I hate having to have a block of `if bar is None` at the top of my functions.

I wonder if there's a library that helps make creating functions with default args easier? Or any tips people have?

View on HN

How about a decorator?

    from functools import wraps

    def defaults(**default_kwargs):  # Decorator factory
        def decorator(func):  # Decorator function
            @wraps(func)
            def wrapper(*args, **kwargs):  # Actual wrapper

                # Iterate over all default keyword arguments
                for name in default_kwargs.keys():
                    # And if not specified in the passed in kwargs use the default
                    kwargs[name] = kwargs.get(name, default_kwargs[name]())

                # Call the actual function
                return func(*args, **kwargs)
            return wrapper
        return decorator
    
    # Note that the default argument value should be a callable to avoid the pitfall with mutable arguments
    @defaults(bar=lambda: [1, 2, 3])
    def foo(bar):
        print(bar)

    foo()

Yeah, exactly. I just hacked together a similar thing here :)

https://gist.github.com/softwaredoug/bbfa2541d48c685908ebd6a...

typing tools won't know what the above decorator does, without some fancy PEP-612 annotations and I'm not sure pep612 can cover that level of automation. I think intent and all is clear for everyone just having the construction done in the method body for the (IMO, should be) seldom case this is needed

Eh. I don't use typing tools for most things because (and I know this isn't a popular opinion even though I believe it's the correct one) they rarely save you from real world problems. They are mostly useful for auto-complete in your IDE, so you have to weigh the time you spend annotating your code to help your IDE auto-complete function signatures vs the time the auto-complete saves you. I think that if you use this as:

    def foo(bar: list):
        print(bar)

that type checking will be happy.

If you use it in conjunction with mypy it definitely saves you from alot of real word issues. In addition, it is also an amazing way of documenting an important element of your code. And annotations van be used by modules (like pydantic) to get even more use. Improvement to anotations have been the biggest improvement in python since 3.6 imho.

What kind of bugs have you encountered in practice that were caught by mypy but not at runtime, manual, or automated testing? I am genuinely curious because in my experience the only time I have seen this is when you either transpose the order or arguments to a function (detected the first time you run it) or if you pass None when an actual value was necessary (detected the first time you run it).

While "weird" when first encountered, once you wrap your head around why this happens, then this is a feature, not a bug.

Both in terms of enhancing one's understanding of python's memory model, and in terms of it being an incredibly useful feature, e.g. for introducing static/persistent variables to a function (without the need for global declarations)

I would not want to see this "fixed"

Common Lisp takes expressions for default arguments, and has the "fixed" behavior; when one wants a persistent value, they do either:

    (let ((my-list '(1 2 3)))
      (defun foo (&optional (l my-list))
        (setf (car l) 'hello)
        l))

or, more often, just make it a special variable of some sort:

    (defvar *my-list* '(1 2 3))
    (defun foo (&optional (l *my-list*))
      (setf (car l) 'hello)
      l))

This seems to work fine, and brings attention to "I have an object that supports mutation, and I'm actually going to mutate it," which is an unusual enough case to be worth calling out.

You are modifying literal data, which is not a good idea.

i mean it should at least be clear, because it’s not like that in most languages. And there could be 2 forms of syntax for default arguments. singletons or each-call

As Python is now, this code:

    def myfun(person = Person()): ...

will evaluate `Person()` only once. If you want it to be evaluated every time, you have to say:

    def myfun(person = None):
        if person is None: person = Person()
        ...

What's being suggested, is to change the language so that the default-arg code:

    def myfun(person = Person()): ...

will evaluate `Person()` on every call. If you wanted it evaluated only once, you would have to say:

    __default_person = Person()
    def myfun(person = __default_person): ...

In either case, it's pretty easy to obtain the other case, and the question is what the best default is. My impression is that "evaluate every time" is both a lot less surprising and more commonly what you want, so it would be the better default.

Right. They were saying they much prefer the current way over the proposed way.

It makes sense if you realize that the function is defined once, and for the function need to be defined, its signature needs to be defined with it. Default arguments are part of the signature, not part of the function body.

It would make sense to pass a callable without calling it, like

    def myfun(person = Person): ...

but if the call occurs in the function signature, it doesn't really make sense that the call would somehow happen again and again. You're not defining a function in the signature; you're specifying a value right there.

The current PEP proposal is to enable the "evaluate at function call time" ability with new syntax, which I think makes the most sense semantically and also doesn't break existing code.

> if the call occurs in the function signature, it doesn't really make sense that the call would somehow happen again and again. You're not defining a function in the signature; you're specifying a value right there.

There are things besides functions and values! They're called "expressions" or "code blocks", and in an impure language like Python they can produce a different value each time you evaluate them. In my second example, "Person()" is neither a value nor a function -- it's an expression that would be evaluated each time the function is run. This isn't that weird; in `while condition: body`, `body` is a block that's evaluated multiple times. It would need to capture variables from the local environment though, which makes it more complicated to implement.

> The current PEP proposal is to enable the "evaluate at function call time" ability with new syntax, which I think makes the most sense semantically and also doesn't break existing code.

Oh yeah I wasn't suggesting breaking existing code. Just that if I was designing a language from scratch, it would probably avoid the Python foot-gun and have default arguments evaluated each time.

> Then this is a feature, not a bug.

> Both in terms of enhancing one's understanding of python's memory model...

In much the same way as having your company bricked by a buffer-overflow ransomware attack does in the C world? I'm not a big fan of elucidation by gotcha.

> [...and] for introducing static/persistent variables to a function (without the need for global declarations)

Except when you are at risk of someone calling with their own variable(s) that do not have the static-singleton semantics that the default would have.

This seems pretty iffy for introducing static/persistent variables to a function. I mean, it can work, but it's semantically very confusing. Parameters are part of a function's interface. A global variable would be much better.

I won't bite on the feature/bug argument, but it's enough of a foot-gun that I have tracked down no less than 3 real world bugs relating to it in large python code bases. Usually when people use an empty list literal as default and shit gets real when it retains values on the next call.

When performing Python interviews I usually try to have that behavior come up in the coding question - not to quiz for trivia so much as to get a feeling for adding on battle-scar bonus points.

In practice, for any non plain-ol-data defaults I just have it default to None and handle it in logic.

Totally understand why it may be useful for introducing a persistent store to a function, but as a matter of taste I like to do that in an explicit decorator if needed.

I agree, this behavior is super useful. If python introduced a static keyword, that would cover some use cases, but wouldn't be as flexible.

There’s a PEP suggested for 3.11 handling this

https://www.python.org/dev/peps/pep-0671/

Sounds like a good idea. Not really a fan of the suggested syntax, though:

  def fn(s, n=>len(s)):

"=>" just kind of bugs me, for some reason. (I love me some bikeshedding.)

Of the alternatives they offer, ":=" kind of looks nicest to me:

  def fn(s, n:=len(s)):

Though it doesn't really line up with the semantics of the existing walrus operator. Also, as the PEP mentions, it could look confusing when mixed with type annotations.

Semantically, the "?=" alternative might be best, since it kind of makes it clear it's a default, and doesn't conflict with anything else in the language:

  def fn(s, n?=len(s)):

Kind of reads like "present? else = ...".

edit: At this point I'd shift from "might be best" to "almost certainly is best". I'm now a zealous "?=" advocate.

I agree. I think python needs some solution to the `def foo(mylist=[])` which is a mistake I see devs make all the time. `foo=>func(a)` is awful. There's too much precedent that `x = f => {}` defines a lambda with x as a parameter. This flips it on its head. `@` would be challenging for the parser. `:=` also does not match the walrus semantics.

`x ?= default` is perfect IMHO. There's already prior art (Makefiles) that this means exactly "present? else = ..." with lazy evaluation of RHS expression.

It also would allow eliminating the "if x is SENTINEL" pattern, where None is a valid non-default argument.

Yeah, the more I look at it, the more I'm a fan of "?=". Kind of seems like the obvious choice at this point.

The only potential downside is that standard, function definition-time default arguments also imply "present? else = ...". But I think of all the options available, this one makes the most sense.

Having mylist stick around is actually useful for making stateful functions. For example, I've often used

  def foo(..., cache={}):
    ...

to implement cached functions, or functions which need an internal cache for whatever reason. Later versions of Python provide an lru cache decorator, but that doesn't help with the internal cache and can be inefficient when you don't want your cache to forget any results.

What about .&= for default arguments?, inspired by Ruby.

I don't think its bike shedding FWIW, Python traditionally eschews fewer keywords and more obvious operators. So it's a reasonable concern.

Yeah, it's probably not bikeshedding when it concerns a very popular programming language's syntax. I just find myself prone to bikeshedding and pedantic naming indecisiveness in general.

The PEP seems entirely focused on bound checking and performing functions on variables in the function definition.

I'm not sure it handles the use case op is talking about.

IE def foo(bar=list()): # this creates a mutable list shard by all calls of the function. Very unlikely something you want.

Maybe the pep would allow for def foo(bar=>list()):

But the pep does not address this use case at all

Even though the PEP doesn't explicitly mention mutable default arguments, I'm pretty sure it's intended for this use case, as well. It should address this issue.

>Function parameters can have default values which are calculated during function definition and saved. This proposal introduces a new form of argument default, defined by an expression to be evaluated at function call time.

> I hate having to have a block of `if bar is None` at the top of my functions.

Doing it the if-statement way is the clearest and easiest for other programmers to follow, therefore it is the best way.

Maybe you could make a function decorator that looked for a special type of default value indicating how to make the default value each time. Similar to `attr.Factory` [1] for `attr.dataclass` annotations.

1: https://www.attrs.org/en/stable/api.html#attrs.Factory

I was thinking this too, if a decorator could make this easier.

I'd also like whatever a solution is to work well with type annotations.

I'm surprised there isn't a pep to improve the situation. (Maybe there is and I haven't found it!)

There is, see my comment here:

https://news.ycombinator.com/item?id=30047413

> Maybe you could make a function decorator that looked for a special type of default value

If you make the functions to supply late-bound defaults arguments to the decorator, its pretty doable; here’s what I just threw together.

    from inspect import signature, Parameter

    def with_deferred_defaults(*args, **kwargs):
        def decorate(fun):
            parameters = signature(fun).parameters
            positional_parameters = [parameter for parameter in parameters.values() if parameter.kind in [Parameter.POSITIONAL_ONLY, Parameter.POSITIONAL_OR_KEYWORD]]
            defaults = list(args)
            kwdefaults = kwargs
            def wrapper(*args, **kwargs):
                args = list(args)
                if len(args) < len(defaults):
                    args = args + [arg() for arg in defaults[len(args):]]
                new_kwargs = {k: v() for k, v in kwdefaults.items()}
                new_kwargs.update(kwargs)
                for i, p in enumerate(positional_parameters):
                    if i >= len(args):
                        if p.name in new_kwargs:
                            args.extend([None]*(1+i-len(args)))
                            args[i] = new_kwargs[p.name]
                            del new_kwargs[p.name]
                        elif p.default is Parameter.empty:
                            raise TypeError('Missing argument: {}'.format(p.name))
                    else:
                        if p.name in new_kwargs:
                            del new_kwargs[p.name]
                return fun(*args, **new_kwargs)
            return wrapper
        return decorate

What about:

    def foo(Bar=Bar):
        bar = Bar()
        ...

Pass in the class or any callable and then call it in the body to make an instance.

Yeah, this is the correct answer and it works with functions or objects.

This is commonly used to implement logging callbacks e.g.

    from collections.abc import Callable
    
    def log_fn(message: str = ''):
        print(f'This is your log: {message}')
    
    def run_read_file(file_to_read:str, log_handler: Callable=log_fn):
        contents = open(file_to_read).readlines()
        for line in contents:
            log_handler(message=line)

The problem with that compared to late-bound parameters is that you have to pass it a callable for non-default uses as well. It's nice sometimes to have a deferred (callable-supplied) default but pass immediate values for non-default.

If this is a class, you can use dataclasses.field with default_factory. However I don't think there is anything analogous for standalone functions.

https://docs.python.org/3/library/dataclasses.html#dataclass...

Functions are in fact objects as well, so there might be some way to do this sort of behavior with decorators.

I would actually contend that "if x is None, x =Bar()" is actually a superior pattern in some/many cases. I used to use "x or Bar()" or "x if x is not None else Bar()" quite a lot, due to its compactness. However I learned that actually sometimes code coverage fails to ensure proper coverage of these ternary assignments. If your default argument is just because you need to initialize a mutable object, ternary works fine. However if the presence or lack of a parameter is an important part of your state space, you probably want explicit coverage.

https://stackoverflow.com/questions/47958166/code-coverage-a...

You can hide the if statements with `bar = bar or Bar() `

This isn't always safe to do. It can break duck typing, because not all falsy values are interchangeable in all situations. For example, consider how numpy methods often have an optional `out` parameter indicating where to write the output. So suppose you are writing a value producing method that takes an `out: Optional[List[int]] = None` to put the produced values into and you handle the `None` case with `out = out or []`. That's a bug! When the user tries to pass in a list to put things into, and that list happens to be empty, you will instead put outputs into a new list.

Anyways, my point is that I can write `if X is None: X = bla` and be confident it will be correct in almost all situations. For `X = X or Y` there are some nasty corner cases where the distinction between `[]` and `""` and `{}` is somehow important, or the distinction between `is` and `==` is important, or different falsy values have different annotations attached to them, or or or....

bar = Bar() if bar is None else bar

All of this is true. Its also weird to me that the evaluation of a boolean expression can return return a value that is neither True nor False (yes I know the value evaluates to true, that's why the trick works).

Which is why I just do `if param is None:... ` and not worry about the few extra lines of code and move on.

The trick does have the nice property of delaying execution for evaluating what the default parameter though. The only other way to do that would involve writing lambdas which is even more code? The sibling comment suggesting a package looks like even more code to write as a user. I think any really elegant fix would have to be done in the language itself.

It's not only not a trick, it's how the language is designed with the intention of doing it like this.

To me it's like complaining that (++a) is a "trick" that lets you use secret temp values that are hard to understand. I mean.. _I guess_, but it's C so it seems like the problem is that you don't like C.

"I hate having to have a block of `if bar is None` at the top of my functions."

Why? This is like someone saying "I don't like declaring my variables in C at the top of the function".

It is a good and readable solution to a somewhat rare problem. Passing mutable objects as default function parameters is a common code smell in Python.

And no, adding a special syntax to an already syntax-heavy language to address this is not good.

Because it’s harder for the user of the API to know what the default is.

It’s boilerplate that distracts from the purpose of the function, this getting in the way of readability.

It’s also just more code prone to a programmer screwing up from time to time, despite how simple it is.

Pythonic way to solve this problem would be to make Bar() a component of another class.

Functions are not a good tool for storing stateful objects in Python. That's why classes exist.

You can do

    def foo(bar=None):
        bar = bar or Bar()

if you know that bar can't evaluate to false in edge cases.

For your example I would do foo(Bar()) when I needed to run it without an existing object. Or make foo a method on the Bar class and do Bar().foo()

In general I just try my best to avoid immutable, or dynamic defaults, but sometimes they're necessary and I make that same if block at the top of the function. If I catch myself copy/pasting the same setup to multiple functions, then it might be time for a class or a decorator.

>I'm curious if anyones found a good solution to Python default args?

Apart from the common solutions (initialize to None and check, etc.) this is exactly what this PEP is about: https://www.python.org/dev/peps/pep-0671/

Python is my go to language but there are a lot of these little hang ups you only know about once you’ve pulled your hair out. I don’t know if this is one technically though, function definitions are set at interpretation so I think the result here is pretty straightforward even if it’s a hang up for some people.

I always point people here for a decent write-up of some "gotchas": https://docs.python-guide.org/writing/gotchas/#mutable-defau... .

What is your list of "these little hang ups you only know about once you’ve pulled your hair out" if you don't mind sharing?

I agree that it's unfortunate that this is the default behaviour. It's just so unexpected when first encountered.

That said, it is not always desirable that a new object is created each time the function is called without the arg. One obvious example is when the default is an immutable object that's either large or expensive to construct.

Although "if bar is None:" is the explicit form, the common idiomatic one is "if not bar:" (unless it's acceptable for bar to be 0).

> unless it's acceptable for bar to be 0

Or an empty list.

Or any other falsish value.

Reply on news.ycombinator.com