Making Python's __init__ method magical | Hacker News Reader

Making Python's init method magical

blog.lerner.co.il

60 pointsreuven40 comments

And here's a more magical version that wraps it with a decorator. If you're gonna do this I kind of like this way better, but I'm pretty sure your average python programmer finds it obscene:

    from pprint import pprint

    def default_init(init_func):
        arg_count = init_func.func_code.co_argcount
        arg_vars = init_func.func_code.co_varnames[1:1+arg_count]
        def wrap_init(self, *args, **kwargs):
            idx = 0
            for arg in arg_vars:
                if arg in kwargs:
                    setattr(self, arg, kwargs[arg])
                else:
                    setattr(self, arg, args[idx])
                    idx += 1

            init_func(self, *args, **kwargs)

        return wrap_init

    class testing(object):
        @default_init
        def __init__(self, a, b, c):
            pass

    x = testing(1, 2, 3)
    pprint(x.a)
    pprint(x.b)
    pprint(x.c)

    y = testing(1, 2, c=3)
    pprint(y.a)
    pprint(y.b)
    pprint(y.c)

[note: I'm aware this does not do even remotely a complete job as a decorator. Use at your own risk, it is a demonstration only.]

Very nice; it does seems far more reasonable to implement this as a decorator than as an explicit function call __init__.

One thing I like a lot better about it in particular is that it doesn't reach into the stack frame, which strikes me as a very messy grab at internals people should never ever touch. The arity and name information from the compiled function object, however, has legitimate use in tooling and I suspect a lot of decorators do this sort of thing.

It's an interesting balance between one being more magical in its use of the system (inspecting live stack frames) and the other being more magical in appearance (having a function with just pass in it actually do something).

nice but it doesn't work if not all kwargs are passed..

Here is another version implemented as a decorator. Actually I’d also like not to use inspect, but didn’t find any nice way to get the kwargs default values without it, any suggestion?

    import inspect
    
    def autoinit(decorated_init):
        def _wrap(*args, **kwargs):
            obj,  nargs= args[0], args[1:]
            names = decorated_init.func_code.co_varnames[1:len(nargs)+1]
            nargs = {k: nargs[names.index(k)] for k in names}
            kwa_keys = decorated_init.func_code.co_varnames[len(nargs)+1:]
            kwa_defaults = inspect.getargspec(decorated_init).defaults
            for k in kwa_keys:
                nargs[k] = kwargs[k] if (k in kwargs) else kwa_defaults[kwa_keys.index(k)]
            for k, v in nargs.iteritems():
                setattr(obj, k, v)
            return decorated_init(*args, **kwargs)
        return _wrap
    
    
    if __name__ == '__main__':
    
        class Foo():
            @autoinit
            def __init__(self, a, b, x=0, y=0):
                self.p = 'Hello, I am set inside the __init__'
                self.s = self.a + self.b + self.x + self.y
    
        f = Foo(1, 2, x=120, y=5)
        for param in 'abxyps':
            print param, '=>', getattr(f, param)
    
        f = Foo(31, 33, x=64)
        for param in 'abxyps':
            print param, '=>', getattr(f, param)
    
        f = Foo(7, 121)
        for param in 'abxyps':
            print param, '=>', getattr(f, param)

Good call. It's amazing how many things you have to be careful of when writing decorators.

Really really minor nitpick, but when talking about languages and their traits these little details can become important for the reader to understand, lest it confuse them about other languages:

> While many people think of __init__ as a constructor, that’s not really the case. Rather, __init__ is invoked on our new object after it has been created

I'm not aware of a language (though I don't claim to know them all!) in which this is not exactly the definition of a constructor. In C++, `operator new` does the allocation, the constructor is called after. In java it's similar. In ruby it's Class#alloc (which is called by Class#new as roughly `ClassName.alloc.tap{|o| o.initialize(*args); o}`)

So the reader is correct to interpret it as a constructor. Constructors initialize the state of an uninitialized object.

In Python you have a "real" constructor you can use: __new__. So I think it's good not to call __init__ a constructor. Unless of course you differ in my opinion that __new__ is a constructor?

I do. __new__ is effectively an allocator, or an interface to it. It allocates an unconstructed object. Then the constructor, __init__ is called on that.

Edited to add: Reading around, it appears that smalltalk has no separation between allocation and construction, having only the equivalent to `operator new`/`Class#new`/`obj.__new`, with initialization also happening in overloads of that. So some confusion may stem from that. I do think the distinction is meaningful and important in languages that separate these two things out, though.

I'll have to check my terminology to understand this better. But I can assure you that most of the programmers who take my Python classes (who tend to come from Java, C#, and C++) are rather surprised that the constructor modifies an existing object, and doesn't actually create the object.

That said, I'm totally willing to believe that neither they nor I understood the difference between an "allocator" and a "constructor." I'll check into this further, both for myself and for the sake of future writing and lectures!

Well, they have a poor understanding[1] of the sequence of events in C++ if they think that the constructor allocates the object there (I won't speak too strongly to Java or C#, since I don't work in them if I don't have to, but I believe that they both work in a very similar way). This would in fact be impossible, since many constructors are usually called for any given object (and all constructors in the object's graph are always called, unlike the more traditional OO languages where the upcall is usually explicit). Also because where an object is allocated is a decision of the caller. It can be allocated with new on the heap, on the stack with a simple declaration, or at an arbitrary memory location with placement new.

In C++ the heap object initialization goes like this:

    Obj *obj = new Obj;

This calls either `Obj::operator new` or the globally scoped `::operator new`, which somehow obtains an area of memory at least sizeof(obj) bytes long and returns it. After that calls to the object's constructors are generated and called.

`operator new` is an allocator and the constructor initializes the raw bytes returned from it.

You can actually do the process explicitly in its entirety like so using placement new, which is a special form that calls the constructors on an arbitrary location:

    Obj *obj = ::operator new(sizeof(Obj);
    new(obj) Obj;

You can't call constructors directly (except with a C++0x feature that lets you call them from a same-level constructor) because of the issue of how they're ordered based on the object graph. They have a precondition of their parent class having been fully constructed.

[1] Note: I don't really mean this as a judgement, understanding of C++ internals are pretty poor in a lot of programmers and for pretty good reasons a lot of the time.

Hi, I'm a C# developer who has done some Java and C++ in the past. You may consider me in the box of people who thinks the constructor creates the object, and the object not existing until the constructor finishes. The allocation of memory is not what I would count as creation of an object, as a model of memory is basically unrelated to the idea of objects and language semantics as I use them. If the object cannot be used, I would consider it not yet created.

I am not trying to argue (on the contrary, I would defer to your expertise), but rather I mean to add a small data point of what perspective developers with my background may have.

In the discussion of C++ constructors, someone usually (IME) says something like, "now remember, it's not a real Foo until the constructor finishes!" because the invariants haven't been set up yet. That could be where the misunderstanding you mentioned comes from.

In Smalltalk, 'new' is a class method which allocates a new instance, but which then has no special access to the internals of that newly created instance. An instance method, 'initialize' is then almost always used to set the initial state (by convention - this is not an actual language feature). Most commonly, initialize is called by new, but this again is just a convention, and a fairly recent one.

Interesting. Yes, it is very similar in both ruby and python, except that the convention is stronger by virtue of being the default behaviour of #new or .__new__ respectively. The placement of the methods (new on the class object and init[ialize] on the instance) is identical.

I really should do something in smalltalk. I only know details about it from reading about it, really, I've never had an opportunity to use it directly, though I go out of my way to understand its influences on more contemporary languages.

My very limited experience working with the Smalltalk language and environment was quite negative, although I hear people sing its praises, so perhaps I just didn't know what I was doing.

That said, it seems to me that many Ruby developers (and to a more limited degree, Python developers) are learning Smalltalk nowadays, for the same reason as English speakers learn Latin: to understand the origins of the language, and thus get a deeper understanding of how it works.

Yes, and while I'm generally of the view that Smalltalk has lessons to teach, of course they aren't all good ones - in particular I sometimes feel that the way construction and initialization is done in Smalltalk has tended to incentivise having getters and setters for every internal variable, and has perhaps contributed to the rather brittle style of program construction that is decried as part of "everything that's wrong with OOP".

Good point. Python's __init__ is closer to Common Lisp's 'initialize-instance :after' except for the fact that Common Lisp provides :initarg to allow make-instance to fully construct an object in most cases.

So, we've got three roles and three names... just let's call the allocator "allocator", the initializer "initializer", and the thing which does both the "constructor", maybe? :)

Here's another magical version that uses a class decorator:

  class Autoinit(object):
      def __call__(self, cls):
          orig_init = cls.__init__
          arg_count = orig_init.func_code.co_argcount
          arg_vars = orig_init.func_code.co_varnames[1:1 + arg_count]
  
          class Wrapped(cls):
              def __init__(self, *args, **kwargs):
                  for pos, var in enumerate(arg_vars[:len(args)]):
                      setattr(self, var, args[pos])
                  for var in kwargs:
                      setattr(self, var, kwargs[var])
                  return orig_init(self, *args, **kwargs)
  
          return Wrapped

  
  if __name__ == '__main__':
      @Autoinit()
      class Test(object):
          def __init__(self, a, b):
              pass
  
      t = Test(1, b=2)
      print t.a, t.b

[Disclaimer: This is my first class decorator. Please correct me if I'm wrong.]

I don't see much value autoassignining args. I do see being useful at times to auto-assign kwargs.

  class Base(object):
      def __init__(self, *args, **kws):
          self.__dict__.update(kws)

  class Foo(Base):
      def__init__("foo", age=30):
          self.name = "foo"
          super(Foo, self).__init__("foo", age=30)

or you can do the same using a decorator for the init.

  def update(init_meth):
      def wrapped(self, *args, **kws):
          init_meth(self, *args, **kws)
          self.__dict__.update(kws)
      return wrapped

  class Foo(object):
      @update
      def__init__("foo", age=30):
          self.name = "foo"

Oh boy, so I started using Python first. I took "reflection everywhere" for granted so hard without even knowing it. I learned Clojure next, and while it had its rough points I never thought, "Man this is really hard!"

Then, I learned some Scala & Java. That's when I realized how for granted I took this thing called "reflection". Especially when it came to ORMs. Holy mackerel. I got over it but it is definitely a huge thing -- for better and worse -- that is so easy in Python that really spoils beginners.

The JVM does have powerful reflection capabilities, even though most programmers don't use it directly.

You do see it used for mocking, ORMs, dependency injection, etc.

CoffeeScript has a nice syntactic sugar for setting properties by prefixing the argument name with "@":

  class Foo
    constructor: (@x, @y) ->
      @z = @x + @y
    setFoo: (@foo) ->

Which compiles to: http://coffeescript.org/#try:class%20Foo%0A%20%20constructor...

Interesting you mentioned CoffeeScript and syntactic sugar, because adding an equivalent "self." in python inits' parameter names doesn't work the same way in CoffeeScript.

    def __init__(self, self.x, self.y):  # fail
        pass

Did you mention this because there is an existing PEP that proposes this functionality?

No, I just thought the way CoffeeScript handles it is pretty slick. I looked around, and it seems like proposals to add something like this to Python did come up [1] [2], but weren't very welcomed.

[1] https://mail.python.org/pipermail/python-dev/2005-July/05456...

[2] http://bytes.com/topic/python/answers/101594-__autoinit__-pr...

I prefer the simpler:

    class A(object):
        def __init__(self, x, y):
            self.__dict__.update((k,v) for k,v in locals().items() if k != 'self')

Seriously, I couldn't finish the article once I saw he wasn't using locals. Is there some hidden gotcha that we are missing here that he's taking care of with his more complicated approach?

Because that would include non-argument locals, if you have any.

I don't like code to casually access the inner dict. Now you have to maintain your `k != 'self'` (ugly in its own right) for each argument you want to exclude from this rule. The explicit version is superior, this is needlessly slow and complicated.

I had a different take on this - it's somewhat inspired by collections.namedtuple but mutable and has some other convenient features for data classes: https://gist.github.com/tomstrummer/8240282

(see the examples at the bottom of the file)

This could also be tweaked to use __dict__ instead of __slots__ so it can act as an expando class which I believe is more similar to what the OP achieves.

Please don't.

    def __init__(self, x, y):
        self.x = x
        self.y = y

is easy to read, easy to maintain, and not even that hard to write.

Sure, there's some magic involved, but it's pretty explicit. If this was standard I would use it.

> but it's pretty explicit

I'm not sure you and I have the same meaning for that word.

This is pretty cool (reminds me of the scala approach to constructors). But please listen to the author's caveats, and never do something like this in a production system; there are few things worse than magic when you're debugging.

For me it's rather common (more than 75%) for the __init__ parameters be other than simple assignments to attributes.

This kind of complexity and the esoterica involved is exactly what terrifies me about Python.

The thing with Python is that you don't need to know this stuff to use the language effectively. Yes, after a while, you'll want to dive into the innards -- but they're optional for most people during their initial time learning Python.

Yes, but the depth of the pool is far from clear when you are considering diving in. :-)

I remember seeing something like this in Matellis' Python Cookbook.

Reply on news.ycombinator.com