Python Black Magic | Hacker News Reader

Python Black Magic

jeffbail.com

19 pointsjbail24 comments

This is not exactly "black magic", it's fairly well-known and explained directly in the official documentation: http://docs.python.org/tutorial/classes.html#private-variabl....

I think developers coming to Python from other languages (like I am from a heavy Java background) find the auto-creation of "shadow variables" pretty interesting. To someone used to Java, being able to access a private member via its shadow variable is a lot like black magic.

Discovering this for the first time in Python seemed pretty cool to me. I really hope this feeling of magic in working with Python never leaves me. I want to remain amazed.

Coming from C++, this is very similar to what all C++ compilers do internally (i.e. name mangling). Python just opens a possibility to access that "mangled" names directly, which should never be necessary unless you are doing some metaprogramming and need full access to class internals.

I learned these kinds of tricks when I started using python 12 years ago, but I have basically never had a need for them. Perhaps my needs are simple, though.

Same here. Like many things in python, the unconventional hack is made just inconvenient enough to persuade most people to rethink their strategy.

Agreed. I can't think of a time when you'd need to use it either (and I wrote the post). But, coming from working in Java for the past few years, it seems pretty cool that you can do it.

You also can do it in java: http://www.fordevs.com/2008/11/reflection-is-mechanism-by-wh... . As a python programmer who never uses "private" variables (I use a single underscore to indicate fields that shouldn't be used), I'm a bit torn as to which private variable abuse method is uglier :)

They are nice to know when debugging -- looking at dir(obj) can lead to things like "where the heck is _foo__attr defined?" if you don't know the __x -> _classname__x mangling.

I learned this trick when I first picked up Python, too. I came from Java and, naturally, started writing Python code in the style I had learned to write Java: Heavily object-oriented, with getters and setters everywhere.

Eventually, though, I stopped trying to program in Java in Python. Once you relax and stop trying to hide your precious instance variables from all of the potentially pernicious or incompetent programmers who might be using your code (usually, in my case, just me), you no longer need tricks like this. And you realize that you never really needed them to begin with.

I'm not advocating using getters/setters in Python. Just pointing out that you can access "private" members. I don't typically use getters and setters in JavaScript either.

Java, on the other hand, kind of forces you into using getters and setters because the POJO is so ingrained into the ecosystem. That said, if you write Android apps in Java, you should avoid getters and setters because they aren't really necessary and they create extra overhead.

One of the first textbooks on PL/1 also stated: "you should avoid using subroutines because they they create extra overhead" ;)

40 years later and we still there ...

And you can turn your member access into called code when you need it:

    class Foo(object):
        @property
        def bar(self):
            print "Running getter"
            return self._bar
        
        @bar.setter
        def bar(self, value):
            print "Running setter"
            self._bar = value

While we are on the black magic thread, are there other meta-programming tricks that people find useful/clever?

I'll share one: import inspect. It's handy to use during debugging.

Forgive my incompetence, but I have never gotten the point of private methods/variables. Could someone please explain why they exist?

Edit: I know it's slightly off-topic.

Private members are generally just to protect your from yourself. If you have an integer that should /always/ be 1-10, it's easier to use a private variable and a setter that ensures the variable is always valid, than to hope all your code follows the rules. (Then also if the rule changes, e.g. to 5-100, you just need to change one function).

There are some other uses and examples, but this is the simplest one i can think of off the top of my head.

I may not be the most qualified person to answer this, but I always thought private methods and variables were for internal use inside of a class while the public methods were what one would actually call with their objects. For example your object had a method foo() which would be better defined as smaller methods which are combined together. Those smaller functions serve no other purpose and therefore do not need to be called by the object directly.

One of the points not discussed in the other replies yet, is that public methods and variables are the interface, the well defined, relatively static way of using a class. The private members are internal implementation details, and may change frequently -- not to mention completely change in style, functionality and so on.

The idea then, is that larger teams can be divided into sub-teams, and work in parallel because the interfaces between components don't change, even if the internals change drastically. Essentially it allows components to be somewhat independent. How this works in practice tho...

Private methods/variables etc are those which are not publicly available. Typically internal state for the object.

Ideally, or at least for purity, all variables should be private and only accessible via get/set methods, thus encapsulation is enforced.

Access specifiers (like "private") are language features used to implement the OOP notion of encapsulation: http://en.wikipedia.org/wiki/Encapsulation_(object-oriented_.... Encapsulation is generally cited as one of the defining features of object oriented programming.

Which is weird, because it's actually a defining feature of modular programming.

Public/private access are used to specify an interface to arbitrary functionality. Other modules can call a public interface, but all private internals can be removed / modified freely.

Yep, plenty of C modules do hacks to block access except through the defined API. Most are content simply to use void pointers in public header files, but some do nasty things like generating a random integer at library initialisation time and xor'ing all returned pointers by that number before returning it. (Eww.)

Arguably, it's a matter of isolation from 'irrelevant' changes. In a function, one wishes to keep the ability to redefine their local variables (perhaps, to implement a faster/smaller-space algorithm) without otherwise affecting the overall program -- when they call other code, that other code should avoid accessing the caller's variables, lest the caller be stuck with precisely those variables and that implementation. If this isolation exists, I have more freedom to vary the implementation.

Likewise, private members of a class are implementation details I may wish to change later -- but I would like some assurance that other components will be isolated from this change, so as not to break.

Edit-add: oh, I of course forgot to mention -- for those not familiar, python permits both :-)

You can access private data in C++ too quite trivially, especially since there's no compile-time encapsulation...

Guido van Rossum on the lack of private members: "We're all consenting adults here."

(And a discussion: http://mail.python.org/pipermail/tutor/2003-October/025932.h... )

Also, Ruby lets you call private/protected methods with Object#send.

Reply on news.ycombinator.com