Python metaclasses | Hacker News Reader

Python metaclasses

ivansmirnov.io

55 pointsaldanor29 comments

Great tool, but not to be put in the wrong hands. The folks at Django got it right. We used it for unifying, simplifying and ensuring consistency of deeply nested configuration data entered using different means. However, in environment with less experience, I have seen "Akira-final-scene-tetsuo" monsters come to life with this.

Another loosely related suggestion: if we could stay away from singleton as examples, I think we would save ourselves a lot of future troubles. ^^

"Great tool, but not to be put in the wrong hands."

I wouldn't quite say this - I wouldn't want to discourage someone who's going to learn them to go ahead, but I'd just say "Only use them when you really need to use them and if you're going to use them make sure you really understand them".

The problem with this is the "wrong hands" are exactly the kind of programmers who are convinced they really need complex abstractions when a much simpler option is available.

I enjoyed reading through the Django ORM code to understand how it worked. Metaclasses in python are fun, but I can rarely justify actually using them for anything other than messing around. Oh but how I wish I needed to use them more!

This is an interesting and fairly well explained post, but a little light on real world examples.

Can anyone maybe suggest a few examples of where this could be used (or is used already?)

We use meta-classes extensively. One of the simpler examples I have used are as follows:

We have a class that has different methods that performs numerical analysis/computations of financial models. These methods need to be tested with different subsets of data. The number of combinations of tests and test-data are huge, whereas the 'testing code' is simple and repetitive. Having these tests in a loop is not useful due to way test errors get reported.

Therefore, we generate a test class using the meta classes, that creates 'test_*' methods that are eventually executed. With such methods in place, we have around 100 tests generated within less than 100 lines of code, without loosing the detailed error reporting in case of failing tests.

Could the same thing be achieved by subclassing your base test class and then overriding setUp to inject the correct data for that combination? I'm pretty sure I'm oversimplifying, but I'm really interested in your method.

In Django, when we add a new field type, you need to remember to test all the integrations (admin, forms, migrations, etc) and that's not always done correctly. It'd be great if there were some easier method than just subclassing a template test class and implementing all tests from the ground up.

I do not think it would work with sub-classing. With the approach we have it is just easier to add the 'new' method name to test into the list argument of the meta class. The meta class would generate a `test` method with the passed in function name.

With the subclass approach you are required to implement a new class and essentially write repetitive code, or reuse existing functions. Reusing existing test methods does not provide good error reporting as I had stated earlier. It is late in the day. Shortly, I will try to post a gist example of what I am talking about, if you are still interested.

Bit late to respond, but yes I would be interested, thanks!

The registry really does seem to be the most common use case:

http://ivansmirnov.io/python-metaclasses/#metaclass-as-a-reg...

This enables things like plugin systems, where you want the act of importing a plugin which defines some objects to load those objects into the available plugin list.

I've written a few code spikes using metaclasses and to be honest, it never worked out well for me. I always found a simpler, clearer way of achieving the same outcome without them and the code that I did write felt dirty.

It's a very powerful language feature. Also very dangerous. It's a magnet for technical debt and obscure bugs.

I've used them to build a serialization/deserialization library for a proprietary protocol. To cut down on needless boilerplate I had the metaclass build setter/getter/en- and decoders from a static member variable. Inspired also by how Django describes its data model.

I've used metaclasses to implement debug logging, where every class method logs its parameters and return values:

https://github.com/bkeroack/elita/blob/27129ea64382275effa99...

The Django ORM or sqlalchemy good starting points. Metaclasses are what make them tick.

Here's one good example: abc (abstract base classes) from Python's standard library -- https://hg.python.org/cpython/file/3.4/Lib/abc.py

I've used it to simulate new language features. It sort of allows you to program the type system.

I've had fun using them to make quasi-orms for CSV and Excel spreadsheet data before.

This seems to still miss the critical mechanism that metaclasses provide to the language.

There are many ways to enforce a constraint from dervied classes to base classes.

Metaclasses are the only* way to enforce a constraint from a base class to a derived class.

(* there's actually another way, but don't do it.)

Well, since you mentioned it... what is the other way? I'm curious. I promise I won't do it.

I like Ruby's object model and uniform access principle much better. You don't have to worry about descriptors and meta-classes when doing metaprogramming. Just a few hook methods and `instance_eval` and you're good to go.

Metaclasses are the monads of python. Everyone seems to want to write about them at some point.

Tangential question, is there anything similar to Moose (Perl) in the Python world?

Caveat: I haven't written any Perl except for an occasional one-liner in a decade.

But I do a fair amount of Python, and a 20 minute cruise through the Moose docs leads me to believe that the most similar thing to Moose in the Python world is, well, Python.

Not quite. One of the biggest features of Moose is it's concept of roles. They're in some senses similar to typeclasses in Haskell and Traits in Rust.

Effectively, they give you access to parametric polymorphism in perl land.

http://modernperlbooks.com/books/modern_perl_2014/07-object-...

True.

Rather than having the kind of parametric polymorphism you get with roles, Python prefers duck typing. And for the practical use of adding some composed, reusable, but orthogonal state and behavior to a class, Python provides multiple inheritance. For type-checking issues where duck typing is not sufficient, Python has Abstract Base Classes.

Now, I am aware that roles are not the same thing as duck typing or multiple inheritance, or ABCs. Not having roles in Python is a design decision. http://legacy.python.org/dev/peps/pep-3133/

But I still think in terms of the ability to use the language to write clean OOP code, Python is closer to Perl5 + Moose, than Perl5 alone is to Perl5 + Moose.

(Not a knock on Perl5 or Moose, between 2002 and 2005, I did a bunch of web and systems programming in Perl, and I wish Moose existed then.)

What's moose?

http://moose.iinteractive.com/en/

TL;DR: Moose is an awesome OO framework for Perl.

I'm afraid that's not enough information for me to understand it, so I can't tell you whether there's something comparable for Python.

This topic has been beat to death. I guess there are more articles explaining metaclasses then people who understand them. There are 11K results for /understanding python metaclasses/ in google.

Reply on news.ycombinator.com