Hacker News Reader

Comment by praptak | Hacker News Reader

I have a mildly surprising snippet which I believe they don't have. It's about interaction of class and instance attributes with behavior of "X += Y". Python expands it to "X = X + Y" but only if X implements "+" but not "+="

    class A:
        a = (1,)  # tuples are immutable and don't have "+="
        b = [1,]  # lists have "+="
   
    obj = A()
    obj.a += (2,)  # this creates obj.a
    obj.b += (2,)  # this modifies A.b

    print(A.a, A.b)

The snippet prints "(1,) [1,2]".

View on HN

Oh, that's a wtf for sure. More to be scared of and more paranoid I'm going to appear to junior devs while reviewing merge requests. If I'm not mistaken there's a chapter in Effective Python about this, where they recommend never using lists or dicts as Class attributes or default values for parameters and instead use factories or use None and then initialize them with default values when a new instance is created.

> If I'm not mistaken there's a chapter in Effective Python about this, where they recommend never using lists or dicts as Class attributes or default values for parameters and instead use factories or use None and then initialize them with default values when a new instance is created.

Ya, this is because the initialized object is stored as the default parameter, and is not initialized every time the function runs.

    def functy(dicty={}):
        print('dicty=', dicty)
        return dicty
   
    output = functy()
    # dicty = {}

    output.update({'key': 'value'})

    functy()
    # dicty = {'key': 'value'}

Thanks for bringing this up, I didn't make the connection to default args.

I think this behavior is actually "default args" but reversed. In case of default args it is the mutability that causes the nasty surprise (ooops, all calls share the same object, which is mutable). In case of the "+=" it is the _immutability_ that causes the surprise - I would not expect "obj.a += ..." to create an instance attribute that shadows the class one.

I think the practical conclusion here is "don't call += on objects that don't support it - because it may work and this is not what you want!)".

Maybe we are looking at it differently. To me, the behavior of "obj.a += (2, )" is the obvious and expected one i.e. I expect a tuple "(1, 2)" to be created and stored in "obj.a" and the class attribute "A.a" is untouched and remains "(1,)". The behavior of the "obj.b += [2,]" for the list is the real gotcha here where it correctly alters the value of "obj.b" but wtf'ingly also alters the value of the class attribute "A.b" which completely alters the behavior of A.

I'm curious if you see it the same way?

> The behavior of the "obj.b += [2,]" for the list is the real gotcha here where it correctly alters the value of "obj.b"

It seems to, but, actually, it doesn't.

> but wtf'ingly also alters the value of the class attribute "A.b" which completely alters the behavior of A.

Actually, all it does is call the __iadd__ method on the list that it is the value of A.b (which modifies that list in-place.)

The value of A.b can, incidentally be accessed via obj.b because obj.b does not exist and the lookup path for instance attributes includes the class.

There are two things going on here there are potential sources of confusion, because they behavior differently in different circumstances: object member access (which can get members from the object or, if they don't exist on the object, from the class) and the += operator, which calls the __iadd__ method on the left value or acts as add and then assign to the name on the left, if that name doesn't reference a value that supports __iadd__.

> The value of A.b can, incidentally be accessed via obj.b because obj.b does not exist and the lookup path for instance attributes includes the class. [...] the += operator, which calls the __iadd__ method on the left value or acts as add and then assign to the name on the left, if that name doesn't reference a value that supports __iadd__.

That is not correct. += will always perform the assignment, hence the classic oddball concatenation:

    >>> v = ([],)
    >>> v
    ([],)
    >>> v[0] += [1]
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: 'tuple' object does not support item assignment
    >>> v
    ([1],)

So `obj.b += [2]` desugars to something along the lines of:

    l = obj.b
    l.extend(2)
    obj.b = l

Hence if you call vars() on the instance or access its __dict__, you will see that the instance does have an intrinsic "b" attribute:

    >>> class A: b = [1]
    ... 
    >>> a = A()
    >>> a.b += (2,)
    >>> vars(a), a.__dict__
    ({'b': [1, 2]}, {'b': [1, 2]})

And if you re-set the attribute on the class, it won't affect the instance, which it would if the instance delegated to the class:

    >>> A.b = [5]
    >>> a.b
    [1, 2]

My intuitions come from C and C++, so "+=" should always modify a single object (in C lingo it would be "lvalue") in place. Storing the result somewhere else than the left side of "+=" is counterintuitive to me.

Side note: the object can be immutable, but then the (variable or attribute) left side should be replaced with a new immutable object.

> My intuitions come from C and C++, so "+=" should always modify a single object (in C lingo it would be "lvalue") in place. Storing the result somewhere else than the left side of "+=" is counterintuitive to me.

I thought of this as though the list A.b was marked static - then any modification of b through any instance of A modifies the shared static member

  >>> class A:
  ...   b = [1,]
  ...
  >>> obj = A()
  >>> obj.b.append(2)
  >>> A.b
  [1, 2]
  >>> foo = A()
  >>> foo.b
  [1, 2]
  >>>

Gotcha, I see where we differ. I've exclusively programmed in Python and since int, str and tuple are immutable in Python, I've always assumed "+=" operators create copies and don't update the object.

> int is immutable

But what does that have to do with assignment?

  >>> a = 3
  >>> a += 4
  >>> print(a)
  7

3 and 7 are immutable, yet a takes on those objects.

Obviously, this is inspired by the C += operator, and behaves accordingly, when it's not being weird.

If you go back to the original comment from the user praptak, you will notice in his example the assignments "obj.a += (2,)" and "obj.b += [2,]". These assignments work exactly as expected no matter which school of thought you come from.

The difference is in how the Class attributes are affected i.e. the values of "A.a" and "A.b".

  # obj.a += (2,) has no impact on A.a since tuples 
  # are immutable and copies are made.
  # This is unexpected from a C/C++ intuition.

  >>> print(A.a)
  (1,)    
   
  # obj.b += [2,] has an indirect impact on A.b
  # since lists are mutable and updates are made in place.
  # This is unexpected for beginners in Python since it
  # deviates from how int, str and tuples behave as "+="
  # can be used for int, str, tuples and lists.
  
  >>> print(A.b)
  [1, 2]

Edit: formatting

> These assignments work exactly as expected no matter which school of thought you come

No, they don't.

obj.b does not exist, and no assignment to it takes place, and there is no “indirect effect”.

A.b exists, and because class attributes can be accessed as if they were members on instances, and because A.b has an implementation of in-place addition, that implementation is called on A.b instead of an assignment to obj.b. That is the only thing that happens as a result of the statement that looks like it might be an assignment to obj.b. The effect on A.b is the only action, not an indirect effect of an assignment.

> The behavior of the "obj.b += [2,]" for the list is the real gotcha here where it correctly alters the value of "obj.b" but wtf'ingly also alters the value of the class attribute "A.b" which completely alters the behavior of A.

It necessarily alters the value of A.b, since `+=` is specifically overridden to update the list in-place. With lists, `a += b` is specifically not an alias for `a = a + b`, instead its behaviour is closer to

    a.extend(b)
    a = a

The issue here is the first part, I absolutely hate this override, and sadly the core team has not learned a thing there as `|=` was overridden the exact same way on dicts when `dict | dict` was added to the language.

Yeah it is a little tricky, but using them as class attributes can actually be handy in some circumstances.

Note to readers unfamiliar with Python: class attributes is a special thing, different than instance attributes.

Definitely one of the trickier things with Python for sure.

There's a similar surprising += 'problem' in JS. If you have the following code:

  async function totalSize(fol) {
    const files = await fol.getFiles();
    let totalSize = 0;
    await Promise.all(files.map(async file => {
      totalSize += await file.getSize();
    }));
    // totalSize is now way too small
    return totalSize;
  }

You get an overly low totalSize. It's caused by 'a += b' expanding to 'a = a + b', and the double-mention of 'a' creating a concurrency issue. If '+=' were a single operation with the right-hand-side being calculated first, it wouldn't be an issue.

I don't think this is a concurrency problem. It's a closure capture problem.

I guess the problem in python is different though, it's about having some data structures that are immutable (like tuple) and others that are mutable (like list)

The problem here is the closure on totalSize and where you're awaiting, totalSize is going to be 0 for all files and then you await for the result of getSize, so in the end totalSize is going to be equal to the size of the last file that completes the getSize call.

To fix this, simply await the result of getSize first, and then add the result to totalSize using +=

> and where you're awaiting

That is the sole source of the issue, and exactly what the comment you're replying to talks about.

The problem is that `a += b` desugars to `a = a + b`, so if `b` is an await you get

    totalSize = totalSize + await file.getSize();

Since javascript evaluates left to right, it first evaluates `totalSize`, gets zero, then `file.getSize()`, then suspends waiting for the result... at which point the handler for the next file can run, doing the exact same thing, repeat for all files.

stupid question incomming: isn't that a '+= await' problem?

Wow, that is surprising.

That must have been a fun one to track down!

I think this example is quite deceptive. When setting obj.a and obj.b, what is your intention? If you want to set the instance attributes, you should have created self.a and self.b in the class constructor.

  class A:
    a = (1,)
    b = [1,]
    def __init__(self):
      self.a = (1,)
      self.b = [1,]

  A.a # this is the class attribute
    (1,)
  A.b # class attribute
    [1]
  obj = A()
  obj.a # instance attribute
    (1,)
  obj.b
    [1]
  obj.a += (2,)
  obj.b += [2,]
  A.a 
    (1,) # still the same class attribute
  A.b
    [1]
  obj.a
    (1, 2) # instance attribute appended
  obj.b
    [1, 2]

I think that's the point is that it does seem rather ambiguous as to the intent. If someone is new to programming entirely, what are they thinking is happening with an operation *on obj.a or obj.b? If they know other languages and are just starting out learning Python, what expectations are they bringing from those languages?

If I had had magic wand, I'd make operations on class attributes from an instance a syntax error, and only allow it from the class name, and perhaps with special syntax like in C++, e.g.:

    A::a = (1,)
    A::b = [1,]

*edit: operation on obj.a or obj.b...

Edit: I understand it now. See comment below.

> # tuples are immutable and don't have "+="

What do you mean? Because it looks to me tuples have "+=":

    a = (1,)
    a += (3,)
    # a = (1, 3)

I get that tuple is immutable so you're actually creating a new tuple and shove it back into variable a, but it does not conflict with "having += operation".

> shove it back into variable a

Python is more of a lisp. The assignment semantics are different to C. Actually, you're letting the name "a" point to the newly created object.

In C, a variable is a memory position you can fill and point to.

In Python, a variable is a name that points to a memory position.

Other than in regard to the way type works, and lexical capture, mainstream Lisps have variable semantics that work more or less like C. The names of lexical variables disappear at compile time; a given a variable may be just a slot in some environment frame (perhaps a stack frame), a machine register, or disappear entirely via constant folding: or any of these three in different areas of the code. In any case, symbols do not "point" at memory positions.

Similarity to C is no accident here, since the lexical scoping concepts in Lisps like Scheme and CL, as well as in C, both trace back to Algol.

Classic Lisp global/dynamic variables may store a global value a "value cell" which is closely tied to the symbol itself (perhaps stored in it).

I would agree that the compilation to machine code might do something different than the high-level mental model. Though the machine code or intermediary code might be interesting to some, the semantics and mental model on a higher level are usually what we reason about. If we don't do it this way, then Lisp doesn't exist and functional programming doesn't exist, because we don't have a machine model that behaves like it.

And in that sense, C assignment and Lisp assignment are different beasts. Mainly in that - again in the semantics of the high-level description - "memory reservation" in C happens when we declare the left-hand side of an assignment and in Lisp it happens when we construct the right-hand side.

Though if you can show me a mainstream Lisp where the assignment works like setting a memory cell and not like a bind, I'm happy to be proven wrong.

If you follow the abstract mental model that the binding construct such as let reserves the memory cells, which assignment and initialization just fill in, you will not misunderstand your programs, except for some bugs that interact with optimization where you have to know when to let go of the model.

The right hand side reserves memory only in the sense that some heap object is allocated (if that is the case), but that is secondary to the assignment; and that is the same as C also, as in:

  f = fopen(...); // right hand allocates stream; pointer moves into variable.

> if you can show me a mainstream Lisp where the assignment works like setting a memory cell and not like a bind, I'm happy to be proven wrong.

The mainstream Lisps clearly separate binding from assignment.

   (let (x)        ;; x refers to a freshly allocated cell (nil-initialized, in Common Lisp)
     ...
     (setq x 42)   ;; cell is clobbered, replacing nil with 42.
     ...)

Some of the terminologly used in the specifications is a bit confused. For instance, Common Lisp says say that:

1. A variable is a "binding in the variable namespace" (Glossary)

2. A binding is an association between a name and a value (Glossary)

3. Yet, the description of SETQ uses language like: "First form1 is evaluated and the result is stored in the variable var1".

4. LET is described like this "let and let* create new variable bindings and execute a series of forms that use these bindings". No binding creation semantics is mentioned for SETQ.

Results can only be stored in storage places; nothing can be stored in an "association between a name and a value", unless that association is actually set up through a memory cell: the name refers to a location and the location holds a value. SETQ isn't described in terms of breaking an old binding and setting up a new one..

In the case of dynamic and global variables, there is a term "value cell" which the Glossary defines: like this: "The place which holds the value, if any, of the dynamic variable named by that symbol, and which is accessed by symbol-value. "

Let's turn our attention to Scheme. R7RS says in 3.1. Variables, syntactic keywords, and regions this:

"An identifier can name either a type of syntax or a location where a value can be stored."

and:

"An identifier that names a location is called a variable and is said to be bound to that location."

"The value stored in the location to which a variable is bound is called the variable’s value."

The next sentence is a kicker, and can be regarded as a criticism of the ANSI CL definition of binding:

"By abuse of terminology, the variable is sometimes said to name the value or to be bound to the value."

If you think that Lisp variables are bindings to values, then according to the Scheme maintainers, you've fallen victim to abuse of terminology. :)

I would read that as not implementing `__iadd__` https://docs.python.org/3/reference/datamodel.html#object.__...

Ah I got it. So it would fallback to __add__ if __iadd__ doesn't exist.

Thanks!

Tuples don't have an in-place addition method (__iadd__), and if the left operand of += is a name that does not contain a value that supports __iadd__, then l += r is a fancy way of writing l = l + r.

But if l does support __iadd__, then l += r becomes l.__iadd__(r).

That's true though kind of an implementation detail. Conceptually x += y is always x = x.__iadd__(y), and x can have its own semantics for what iadd does. Conventionally that is that mutable objects modify themselves in place and return themselves, while immutable objects create a new object and return that.

This behavior for list and tuple is familiar from regular local variables. Together with the fact that obj.x = ... always means "create or rebind an instance variable" never "rebind the A.x class variable", I think the example is less surprising.

obj.b += (2,) creates obj.b too, it's just that obj.b still points to the same object as A.b.

In Python 3.8 the tuple is in fact changed:

  >>> print(obj.a, obj.b)
  (1, 2) [1, 2]

What happens to `obj.a`/`obj.b` is not the point of the example, you should be looking at `A.a`/`A.b`.

Is that like a static member? Truly, dynamic languages are God's curse upon the Earth.

It refers to the class, not the instance.

I think this is more or less covered by "▶ Class attributes and instance attributes"

Reply on news.ycombinator.com