On the Beauty of Python's ExitStack (2015) | Hacker News Reader

On the Beauty of Python's ExitStack (2015)

rath.org

135 pointspolm2339 comments

I'd go a little further and say context managers are underused on the whole. They're a great tool and aren't used often enough. ExitStack enables using context managers in more situations.

Problem with context managers is that they add up indentation really, really fast. I guess TFA is at least a partial remedy to this though!

TFA?

The Fine Article, as posted by OP

You can do multiple in a single with statement these days.

This is an often overlooked feature of context managers; to clarify, this is a perfectly normal example:

    with open("foo.txt") as foo_file, open("bar.txt") as bar_file, open("baz.txt", "w") as baz_file:
        baz_file.write(foo_file.read() + bar_file.read())

I think you can also do something like this if you want it to be readable

    with (open("foo.txt") as foo_file,
          open("bar.txt") as bar_file,
          open("baz.txt", "w") as baz_file):
        baz_file.write(foo_file.read() + bar_file.read())

C# has the using-statement + IDisposable which is similar context managers in python (except no __enter__).

C#8 then allowed using-statements that do not require blocks, which then are disposed when leaving the current block.

Pre C#8, you had to have a block, similar to how with-statement requires a block aka indentation

    using (var res = OpenSomeResource()) {
      res.doSomething();
      // res.Dispose() implicitly called
    }

Now you can have this (which is a bit like a go-style Defer but for blocks)

    using var res = OpenSomeResource();
    res.doSomething();
    // res.Dispose() implicitly called at end of current block

Of course, you have to take the potentially longer lifetime into account. This way looks a lot like C++s RAII with constructors and destructors (except the destructor here is the IDisposible.Dispose implementation).

It's basically what all variables in Rust do, with customizable destruction by implementing the Drop trait.

Interesting, I think that would make sense for Python.

    def do_thing():
        with open('foo', 'r') as f
        with a_lock
        f.read()
        frobnicate()

I guess the issue is that there is less of a clear idea of scope in Python, what does the above construct do at a module level? What does it do inside another with block? A lot of special rules that don't conveniently fit into something else unlike what I suppose C# has.

You can get that syntax right now using ExitStack.

You can get something in that general area yes.

This is actually vaguely reminiscent of auto release pools in Cocoa.

Wouldn't it be more natural to do this sort of thing with decorators?

For trivial examples, sure. And there are a lot of those in practice, so it definitely applies sometimes. But decorators aren't really usable when you have any intermediate steps that need to be done, like

    def do_thing():
        with a_lock
        check_f_cache_while_locked() # like this
        with open('foo', 'r') as f
        populate_cache(f)

Though I very much like that this exists, and the documentation for contextlib describes its need and use very well[1], I do want to point out that this has always been possible with a tiny bit of recursive glue code like this[2], which works fine even in python 2:

    @contextmanager
    def multi(*cms):
      if len(cms) > 0:
        with cms[0]:
          with multi(*cms[1:]):
            yield
      else:
        yield

    # use with 0..N items, it behaves the same
    with multi(...):
      pass

Prior to finding ExitStack, I've frequently seen people either give up and do something obviously wrong, or try to manually manage their own stack of __enter__ and __exit__ funcs (which works, but exception handling is non-obvious and often loses useful info). Personally I prefer the recursive route, since it very obviously behaves the same as if you wrote the nesting out by hand. It even retains accurate stack traces "naturally", unlike most non-ExitStack tactics I've seen through the years. ExitStack's stack-correcting tactic is... a bit more "interesting"[3], though AFAIK it achieves roughly the same stack trace in the end.

Since ExitStack exists, and is likely more efficient (by how much, I have no idea), it's probably a better choice for most situations. And it's just broadly a bit nicer for things that yield resources.

It's always worth browsing stdlib documentation to find gems like this, there are quite a few neat things in most languages that seem fairly unknown.

[1]: https://docs.python.org/3/library/contextlib.html#supporting...

[2]: https://repl.it/@Groxx/TepidRosybrownLocatorprogram#main.py

[3]: https://github.com/python/cpython/blob/master/Lib/contextlib...

I always thought it was a shame that `nested`[0] (the stdlib equivalent to your `multi` above) was deprecated in 2.7 and, ultimately, removed in favor of `ExitStack` -- despite its flaws, I think it was more beginner-friendly.

[0]: https://docs.python.org/2.7/library/contextlib.html#contextl...

Yeah, and the source for `nested` is a lot easier to follow than ExitStack. Conceptually it's the same as mine, but with a loop rather than recursion, and it'll yield values (not hard to do with a recursive one, I just didn't feel like it): https://github.com/python/cpython/blob/2.7/Lib/contextlib.py...

Seems like the __new__/__init__ issues exist with ExitStack as well, unlike with `with a, b, c`? So it's not really resolved / improved behaviorally, though ExitStack does have a few new uses (e.g. managing non-context-manager resources).

Haven't used it since I'm just hearing about it now, but seems like the big win to using ExitStack is that you're not required to acquire all the resources at the same time. So you can acquire one resource, do some stuff, then acquire another resource based on something that is dependent on the first without more indentation.

I agree that the recursive solution is nice and simple. We've also used it at work.

However, you can run into "maximum recursion depth exceeded" issues if the number of context managers is extremely large. ExitStack handles this situation nicely [1].

Note that I'm not saying that using so many context managers is a good idea ;-) In our case, using ExitStack (which also has a backport to Python 2 [2]) was the simplest fix though.

[1] https://github.com/freininghaus/notes/blob/master/misc-noteb... or https://nbviewer.jupyter.org/github/freininghaus/notes/blob/...

[2] https://contextlib2.readthedocs.io/en/stable/#contextlib2.Ex...

The comparison with other languages seems to miss the fact that the idiomatic approach in Rust and C++ is RAII, which does let you put the cleanup code next to the allocation code.

..and that Java has, like the object oriented language that it is, an AutoCloseable interface which you can implement and then use in blocks, much like with this Python construct.

In fairness, Java's AutoCloseable interface is missing some of the niceties of ExitStack, as with C++'s destructors. People have made libraries for both that attempt to close the gap.

I write a lot of Python and didn't know about this despite it being a 2015 blog post! The real kicker for me is:

* [ExitStack] easily scales up to many resources (including a dynamic number that's acquired in a loop)

In particular ExitStacks are useful when you have 0 or 1 resources. It allows a sort of null-context manager for when the number of resources you have to allocate might be 0.

Do be aware that https://docs.python.org/3.9/library/contextlib.html#contextl... also exists as of 3.7 -- you can do

  ctx_mgr = foo() if bar else nullcontext()
  with ctx_mgr:
      ...

if you like.

Or even, from Python 3.8:

    with (ctx_mgr := foo() if bar else nullcontext()):
        ...

Here's a good example IRL.

https://github.com/vertexproject/synapse/blob/master/synapse...

It grabs a transaction on an arbitrary number of databases, then yields.

It's ExitStack.enter_context(), not enter().

https://docs.python.org/3/library/contextlib.html#contextlib...

In Haskell, this is done with the Managed monad or ResourceT if you're feeling impure.

I never knew about ExitStack. It reminds me of Haskell's ResourceT.

It's beauty in the sense that it is addressing a language deficiency ("destructor" behavior, awkwardness of the "with" construct) with a library, sure, but this is ultimately working around a gap in Python itself.

I'd argue the context manager concept is better than destructors, because it's more explicit.

BTW, Python does have the `__del__` special method. It's just that almost all usage of it would be better implemented via context manager.

I think the idiomatic way is with weakref https://docs.python.org/3/library/weakref.html

Particularly note "finalize provides a straight forward way to register a cleanup function to be called when an object is garbage collected"

That's a little snarkier than I intended. It is good to share this method of handling what is otherwise very awkward code. It just is unfortunate the language itself doesn't have constructs that lend to proper syntactically scalable resource handling other than nested "with" blocks.

Agreed. I like Python, and I think ExitStack is a fantastic way to manage resources. However, that is because it most closely emulates RAII in C++.

The limitation that I still run into is when a class needs to have some cleanup code added to it. In C++, you just add a destructor. In Python, you are forced to choose between non-deterministic release of resources through __del__, or changing the interface by requiring users to use a "with" block. Neither of those is a particularly satisfying option.

The need to add cleanup code is a change to the interface. If you don't change the interface, you've left yourself open to bugs around accidentally using an instance after it's been cleaned.

If you really want to keep the same interface, add a layer of abstraction. The current interface becomes a wrapper around the underlying context manager.

Exactly. The language design of Python means that the addition of cleanup code is an API change. The language design of C++ means that the addition of cleanup code is not an API change.

I actually appreciate that API change, as it makes my use of resources more explicit. However, you might be interested in using ``weakref.finalize`` to keep that C++ style.

https://docs.python.org/3.6/library/weakref.html#comparing-f...

imo the main benefit to context manager is reporting errors during cleanup while the main benefit of RAII/destructors is composability.

Reply on news.ycombinator.com