The unknown features of Python's `operator` module | Hacker News Reader

The unknown features of Python's `operator` module

martinheinz.dev

88 pointsfeross56 comments

The article makes a mistake in comparing lambda vs operator.add performance. The lambda example is recreating the function each time.

    $ python -m timeit "(lambda x,y: x + y)(12, 15)"
    10000000 loops, best of 3: 0.124 usec per loop
    $ python -m timeit -s "from operator import add" "add(12, 15)"
    10000000 loops, best of 3: 0.0565 usec per loop

Instead, it should create a function in the setup and use that for the loop:

    $ python -m timeit -s "func = lambda x,y: x + y" "func(12, 15)"
    10000000 loops, best of 3: 0.0779 usec per loop

Generally it is un-pythonic to assign lambda to a variable. Lots of linters will complain if you do this.

So do a "def" instead, it can be even inlined.

def add_fun(x,y): return x+y

But I wouldn't mind irritating the linter with an assigned lambda. I fail to see what's the problem with it

But if you’re doing that, why not just import the identical function?

One major point of OP was that the functions defined in operators are faster. Grandparent comment argued that there is a measurement error that invalidates this point, since properly defined lambdas (or defs) are just as fast.

Of course the point regarding the simplicity of importing a function instead of defining it still stands.

I personally wouldn't mind defining those functions as they are very immediate. So from the article I would take only the part about the more exotic "operator".

The timings are certainly closer, but the built-in function imported from operator is still faster than the user-defined function. Not a big deal if you're doing simple stuff, but if this code is in a hot loop that 30% can make a measurable difference.

Lambdas don't know their name and weaken debug info.

It seems like an implementation issue of the interpreter no? There’s no reason lambdas _have_ to be this way.

lambdas don't have names, how could the interpreter possibly improve in this regard?

JS arrow functions don't have names either, it's inferred from the left hand side variable name. Can't Python do something similar?

If you assign a lambda to a variable, you might as well use a def. A more typical use of a lambda is as an arguments to a function call. What would be the name of, say, the first argument to map(lambda x: x*2, numbers)?

Linters will complain about all sorts of nonsense, and I'll have to continue to remind people that pep8 was only supposed to be guidance, and never something that was enforced.

In short: being picked up by a linter does not mean that something is un-pythonic.

The reasoning that I’ve seen for this is that you lose context information (file, line, etc) when using lambdas. Is that it? If so it seems like a an interpreter implementation issue.

The point is that the function is created only once, like when you pass a lambda to map/filter/reduce or any function that takes a callback. The fact that it is assigned here is just a neccessary step for the timeit command to do the right thing

Is that a mistake, or is it just the reason why the lambda style is slower? If you're assigning the lambda, then as @sharikous mentions below, you might as well just be using a def, which is exactly what is done in the operator module: https://github.com/python/cpython/blob/3.9/Lib/operator.py#L...

For someone familiar with the operator module, there is a final small argument for using it instead of defining your own, if the result will be the same: that's a definition that doesn't have to be understood by the reader.

You're reading the pure-python version of operator.py. You get the builtin versions of the same functions - not the python functions when you use it - see the import of _operator further down the file.

Also verify the type info on the object here:

  In [1]: import operator
  In [5]: operator.add??
  Signature: operator.add(a, b, /)
  Docstring: Same as a + b.
  Type:      builtin_function_or_method

'operator' actually maps to C implementations if possible; you're still calling Python functions, but the implementations in 'operator' should be a bit faster than defining your own in pure Python (or the pure Python implementations that get used if the _operator C module is not available for some reason): https://github.com/python/cpython/blob/3.9/Modules/_operator...

I mean, it's the reason the benchmark is slower, because the benchmark is comparing doing 2 things with only doing one of those things -- this is virtually tautological.

So I agree that it is why it's slower, but I also agree that it's a mistake. This is the "hard part" about synthetic benchmarks. This one is measuring something people rarely do: creating an anonymous function and calling it in a tight loop.

It's extremely unlikely you're ever going to run into a situation where creating this lambda is in a hot path, and if you do the thing you did wrong is very unlikely to be whether you used a standard library function or one you defined yourself. It's that you have a loop where you did almost nothing but create a lambda every iteration instead of, say, passing it to a map/fold function or whatever.

A typical use case for lambdas is passing them as arguments to other functions, which implicitly assigns the lambda to some local variable. Using "def" instead would be unlikely to be cleaner.

Maybe you still want to know how long it takes to create a lambda or import from operator, but if such a function is being called more than a few times (and if not, why are we concerned about performance?), execution time will dominate other factors.

From the article:

  (lambda x, y: x ^ y)(7, 10)
  
  from operator import xor
  xor(7, 10)

> Clearly the second option is more readable.

Clearly, the readable option here is ”7 ^ 10”. The examples are a bit contrived, so readability can’t really be gauged from them.

Can't say I agree with you. Off the top of my head, I couldn't remember what the symbol for xor was (since it hasn't come up once in many years of my programming), but I certainly do know what the word xor stands for.

Yes, articles that explain features with simple examples are almost always full of contrived examples. It's unfortunate that people nitpick them, because the alternative is long and complicated examples that can't be understood at a glance.

If you want to do a parity check on a string:

  parity = reduce(xor, map(ord, str))%2

Except Python3 took reduce away in favor of more boilerplate. And yes, there are multiple alternative implementations of the above.

> parity = reduce(xor, map(ord, str))%2

"% 2" will not calculate the parity of the reduced value; it just extracts the least significant bit.

Can you show me an integer whose parity differs from its least significant bit?

Maybe a stupid question but I've never learned much about parity and this sounds like a good opportunity. Can you give some definitions? Are you talking about error checking parity or 'evenness' parity?

I'm guessing the last one since that's what fits your LSB comment(when using 'evenness' not 'oddness'). But in that case I'm not sure your example is doing, my python might be too rusty but is it something like: get the unicode representation(int) of each character in a string, xor the results and get the LSB?

What I meant by parity was 'evenness' -- but the ultimate purpose was to explain the utility of the xor function provided by the operator module.

But, the more useful concept is parity for error detection, which my example clearly fails at.

Note that these two are different:

https://en.wikipedia.org/wiki/Parity_bit

https://en.wikipedia.org/wiki/Parity_(mathematics)

"parity check on a string" tends toward the former interpretation of "parity". You don't do parity checks by looking at only the least significant bit of every code word.

Ah, I understand now. You're right, an actual parity check should reduce the bits of the final result.

> It's unfortunate that people nitpick them, because the alternative is long and complicated examples that can't be understood at a glance.

If the examples don’t show cases where the feature even arguably is the best option, they’re bad examples of the feature. I don’t think that’s nitpicking.

The operator module is not something most python developers will ever need. It's a collection of very low-level tools that will rarely, if ever, be the right choice in a small code snippet. Can we even talk about the existence of these functions in an easy-to-read blog post, if the only 'right way' to use them is a full-blown DSL parser or whatever? Or, can we trust the reader to not run home and refactor their arithmetic to

  #this used to be y = 2 * x + 1 until I read a sweet blog
  y = operator.add(1, operator.mul(2, x))

I really don't see any danger of that, myself.

I honestly think reduce is way more confusing to read than comprehensions or even just a plain for loop, even as someone who has done a lot of functional programming.

If reduce is the outermost function, I might agree. I do prefer the explicitness of foldl and foldr, personally. But, this is the problem with simple examples. Folks just can't help but nitpick every damned thing. Are we adding value to the conversation about the operator module?

> parity = reduce(xor, map(ord, str))%2 Except Python3 took reduce away in favor of more boilerplate. And yes, there are multiple alternative implementations of the above.

You can import functools.reduce to get it back but maybe thats what you meant with more boilerplate

I've been a weirdo with my Python code recently, and tend towards a more functional programming style. So, the `operator` module is essential for doing some cools things with maps.

For instance, I'm writing a library to simulate the behavior of vectors in R, and that means vectorized operators (the operators runs the length of two paired vectors). All you need to do is make a special tuple subclass that maps the operator along another tuple (vector) and returns your vector subclass:

https://github.com/shanedrabing/pyrat/blob/e0049213152f615e4...

Anyways, this is all to say that I love `operator` and try my best not to use lamdbas much anymore (except where they are honestly simpler).

I try that, but it makes tracebacks harder frequently. Adjusting my programming style based on error message availability is a pet peeve of mine.

In this article, some listed operators are:

    operator.length_hint
    operator.itemgetter
    operator.countOf

What's the (historical?) reason the operator module uses different casing standards?

The operator module is very old: https://github.com/python/cpython/blob/ea4d3f0804a3aad5afaa6...

PEP8 was introduced in 2001: https://www.python.org/dev/peps/pep-0008/

typically in CPython its just a sign that the code is fairly old, or at times (as with the unittest module), that an API was ported from CamlCase language at least partially.

The existence of methodcaller confirms for me the good choices made in the design of Lua.

Tables are simple key-value maps, with an optimization for dense arrays, while symbols are strings with a parser-enforced grammar. A method call is syntax sugar: table:method(...) is converted into table.method(table, ...).

This means that "methodcaller" is simply table[string](table, ...), which I've used in a few places to good effect.

Except now you can't have advanced methods on your container types without intermingling the keys of methods and data.

This is why javascript moved away from their "super user-friendly" concept of objects-as-maps towards having a real `Map` type - because mixing data and program can get weird (and if it's user-controllable data, dangerous).

I suppose the question is - is it possible for a table in lua to contain separate entries for the string "foo" and the symbol "foo", or will one happily overwrite the other?

I thought the main reason for `Map` was mainly for non-string keys?

I'm not sure I've observed a lot of folks in the JavaScript world using Map. I agree there are occasions when it's very useful, but for my money those have more to do with cases where you need non-string keys. For everything else, as far as I can tell, Objects remain entirely idiomatic despite their idiosyncrasies.

That's interesting - I must have just been lucky in what I've encountered ES-wise as of late. It relieved me greatly to no longer have the horror of mixing user data and method/"special" names.

I love

  { __proto__: null, foo: 'bar' }

That creates a clean prototype-less map/record object. I always wish they would add special literals to create such objects.

oh, this is fascinating. I might have to add this to my bag of tricks.

In Lua terms, the question doesn't even make sense. Symbols are just strings which you can type without quote marks using the . accessor, table.foo is syntax sugar for table["foo"].

I have no idea what you mean by advanced methods on container types though. I find the simplicity of Lua tables liberating rather than constraining.

Lua tables are a Map type, anything except nil can be a key, and every key has a value: if you haven't assigned something to that value, it's nil. There's no separate "object" and no separate "symbol", those patterns are implemented using tables and strings. Even global environments (you can have as many as you need) are just tables.

By "advanced methods" I mean, for instance, python's `.update(other)` method (which allows you to merge two dicts/maps), or `.pop(key)` (which allows you to remove & return a value at the same time).

Having such methods on a table which you're still expecting to use as a container sounds like it would allow a attacker to e.g. overwrite your methods if you're processing user-controllable data.

`methodcaller` (https://github.com/python/cpython/blob/6a6bcf16370beff2e0d1a...) is just syntax sugar around `getattr` + a function call which is just syntax sugar + default implementation of `__getattr__` which is basically exactly what it looks like your lua example is (I'm not that familiar with lua).

Actually, I didn't even know about `methodcaller` till I read this blogpost, so I would have implemented your example in python as: `getattr(table, string)(...)`.

I've used one or two of these before, would be interested in more patterns that use, methodcaller (I think in the example given str.strip would have worked too).

In my experience methodcaller is an occasionally-useful "mid-refactoring feature".

That is, sometimes when I'm making a largeish modification there's a point in the commit series where the natural way to express what's going on involves using methodcaller, then by the time the work is finished the use has gone away again.

methodcaller can be useful in functional programming, like

  close_file = methodcaller('close')
  exhaust(map(close_file, files))

where exhaust is a function that consumes an iterator. Obviously this is a contrived example, but it's a weird transformation -- it turns map from 'call this function on each of a sequence of arguments' to, roughly 'call each of this sequence of functions on a fixed argument'

exhaust((file.close() for file in files)) works.

Like I said, contrived example. Though, it's a specific example of a more general pattern where you have a bunch of objects that you need to send a message to. You might use this for event handling in the guts of a game engine.

I believe exhaust(file.close() for file in files) should work as well, no need for the extra parens.

It’s higher order for the sake of higher order, and unreadable as hell when you return to five full pages of this nonsense a year later. These getter-caller generators are cool to show off, but irl the most naturally readable form is either a plain anonymous function, or a special form suggested by a sibling commenter.

Both by a compiler easily readable by a human unreadable out of purely functional languages by design this high fashion came.

Sometimes you need to do higher order stuff (or, maybe you don't? But it exists for good reason)

Do you think it's better to use a conceptually difficult example to showcase a high order function? Or do you think it's better to show an example where the reader can accurately understand the concept and focus on what the function does?

Reply on news.ycombinator.com