51

It's typical to require for some task multiple objects which have resources to be explicitly released - say, two files; this is easily done when the task is local to a function using nested with blocks, or - even better - a single with block with multiple with_item clauses:

with open('in.txt', 'r') as i, open('out.txt', 'w') as o: # do stuff

OTOH, I still struggle to understand how this is supposed to work when such objects aren't just local to a function scope, but owned by a class instance - in other words, how context managers compose.

Ideally I'd like to do something like:

class Foo: def __init__(self, in_file_name, out_file_name): self.i = WITH(open(in_file_name, 'r')) self.o = WITH(open(out_file_name, 'w'))

and have Foo itself turn into a context manager that handles i and o, such that when I do

with Foo('in.txt', 'out.txt') as f: # do stuff

self.i and self.o are taken care of automatically as you would expect.

I tinkered about writing stuff such as:

class Foo: def __init__(self, in_file_name, out_file_name): self.i = open(in_file_name, 'r').__enter__() self.o = open(out_file_name, 'w').__enter__() def __enter__(self): return self def __exit__(self, *exc): self.i.__exit__(*exc) self.o.__exit__(*exc)

but it's both verbose and unsafe against exceptions occurring in the constructor. After searching for a while, I found this 2015 blog post, which uses contextlib.ExitStack to obtain something very similar to what I'm after:

class Foo(contextlib.ExitStack): def __init__(self, in_file_name, out_file_name): super().__init__() self.in_file_name = in_file_name self.out_file_name = out_file_name def __enter__(self): super().__enter__() self.i = self.enter_context(open(self.in_file_name, 'r') self.o = self.enter_context(open(self.out_file_name, 'w') return self

This is pretty satisfying, but I'm perplexed by the fact that:

  • I find nothing about this usage in the documentation, so it doesn't seem to be the "official" way to tackle this problem;
  • in general, I find it extremely difficult to find information about this issue, which makes me think I'm trying to apply an unpythonic solution to the problem.

Some extra context: I work mostly in C++, where there is no distinction between the block-scope case and the object-scope case for this issue, as this kind of cleanup is implemented inside the destructor (think __del__, but invoked deterministically), and the destructor (even if not explicitly defined) automatically invokes the destructors of the subobjects. So both:

{ std::ifstream i("in.txt"); std::ofstream o("out.txt"); // do stuff }

and

struct Foo { std::ifstream i; std::ofstream o; Foo(const char *in_file_name, const char *out_file_name) : i(in_file_name), o(out_file_name) {} } { Foo f("in.txt", "out.txt"); }

do all the cleanup automatically as you generally want.

I'm looking for a similar behavior in Python, but again, I'm afraid I'm just trying to apply a pattern coming from C++, and that the underlying problem has a radically different solution that I can't think of.

So, to sum it up: what is the Pythonic solution to the problem of having an object who owns objects that require cleanup become a context-manager itself, calling correctly the __enter__/__exit__ of its children?

Improve this question
4
  • 4
    I would say the solution with ExitStack is quite Pythonic. Commented Aug 6, 2018 at 5:45
  • @BrenBarn: nice to know, but I'm still a bit scared about the fact that there's only one mention about this solution in a random blog, and not in the official documentation, for what I think would be a quite a common problem. That's what leaves me perplexed. Commented Aug 6, 2018 at 9:11
  • 3
    I'm not sure why you expect this to be in the official documentation. In general the official documentation only documents how things work, not what they're used for. There are tons of common problems whose solutions aren't explained in the official documentation. Here is a related question, where in a comment to his answer Martijn Pieters also suggests subclassing ExitStack for what looks like a related purpose. Commented Aug 6, 2018 at 19:14
  • 1
    A context manager that is designed to make it easy to programmatically combine other context managers and cleanup functions, especially those that are optional or otherwise driven by input data. I feel like the docs suggest that the ExitStack solution is perfectly Pythonic. Commented Aug 9, 2018 at 18:21

7 Answers 7

Reset to default
23
+250

I think contextlib.ExitStack is Pythonic and canonical and it's the appropriate solution to this problem. The rest of this answer tries to show the links I used to come to this conclusion and my thought process:

Original Python enhancement request

https://bugs.python.org/issue13585

The original idea + implementation was proposed as a Python standard library enhancement with both reasoning and sample code. It was discussed in detail by such core developers as Raymond Hettinger and Eric Snow. The discussion on this issue clearly shows the growth of the original idea into something that is applicable for the standard library and is Pythonic. Attempted summarization of the thread is:

nikratio originally proposed:

I'd like to propose addding the CleanupManager class described in http://article.gmane.org/gmane.comp.python.ideas/12447 to the contextlib module. The idea is to add a general-purpose context manager to manage (python or non-python) resources that don't come with their own context manager

Which was met with concerns from rhettinger:

So far, there has been zero demand for this and I've not seen code like it being used in the wild. AFAICT, it is not demonstrably better than a straight-forward try/finally.

As a response to this there was a long discussion about whether there was a need for this, leading to posts like these from ncoghlan:

TestCase.setUp() and TestCase.tearDown() were amongst the precursors to__enter__() and exit(). addCleanUp() fills exactly the same role here - and I've seen plenty of positive feedback directed towards Michael for that addition to the unittest API... ...Custom context managers are typically a bad idea in these circumstances, because they make readability worse (relying on people to understand what the context manager does). A standard library based solution, on the other hand, offers the best of both worlds: - code becomes easier to write correctly and to audit for correctness (for all the reasons with statements were added in the first place) - the idiom will eventually become familiar to all Python users... ...I can take this up on python-dev if you want, but I hope to persuade you that the desire is there...

And then again from ncoghlan a little later:

My earlier descriptions here aren't really adequate - as soon as I started putting contextlib2 together, this CleanupManager idea quickly morphed into ContextStack [1], which is a far more powerful tool for manipulating context managers in a way that doesn't necessarily correspond with lexical scoping in the source code.

Examples / recipes / blog posts of ExitStack There are several examples and recipes within the standard library source code itself, which you can see in the merge revision that added this feature: https://hg.python.org/cpython/rev/8ef66c73b1e1

There is also a blog post from the original issue creator (Nikolaus Rath / nikratio) that describes in a compelling way why ContextStack is a good pattern and also provides some usage examples: https://www.rath.org/on-the-beauty-of-pythons-exitstack.html

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

13

Your second example is the most straight forward way to do it in Python (i.e., most Pythonic). However, your example still has a bug. If an exception is raised during the second open(),

self.i = self.enter_context(open(self.in_file_name, 'r') self.o = self.enter_context(open(self.out_file_name, 'w') # <<< HERE

then self.i will not be released when you expect because Foo.__exit__() will not be called unless Foo.__enter__() successfully returns. To fix this, wrap each context call in a try-except that will call Foo.__exit__() when an exception occurs.

import contextlib import sys class Foo(contextlib.ExitStack): def __init__(self, in_file_name, out_file_name): super().__init__() self.in_file_name = in_file_name self.out_file_name = out_file_name def __enter__(self): super().__enter__() try: # Initialize sub-context objects that could raise exceptions here. self.i = self.enter_context(open(self.in_file_name, 'r')) self.o = self.enter_context(open(self.out_file_name, 'w')) except: if not self.__exit__(*sys.exc_info()): raise return self
Improve this answer

Comments

7

As @cpburnz mentioned, your last example is best, but does contain a bug if the second open fails. Avoiding this bug is described in the standard library documentation. We can easily adapt the code snippets from the ExitStack documentation and the example for ResourceManager from 29.6.2.4 Cleaning up in an __enter__ implementation to come up with a MultiResourceManager class:

from contextlib import contextmanager, ExitStack class MultiResourceManager(ExitStack): def __init__(self, resources, acquire_resource, release_resource, check_resource_ok=None): super().__init__() self.acquire_resource = acquire_resource self.release_resource = release_resource if check_resource_ok is None: def check_resource_ok(resource): return True self.check_resource_ok = check_resource_ok self.resources = resources self.wrappers = [] @contextmanager def _cleanup_on_error(self): with ExitStack() as stack: stack.push(self) yield # The validation check passed and didn't raise an exception # Accordingly, we want to keep the resource, and pass it # back to our caller stack.pop_all() def enter_context(self, resource): wrapped = super().enter_context(self.acquire_resource(resource)) if not self.check_resource_ok(wrapped): msg = "Failed validation for {!r}" raise RuntimeError(msg.format(resource)) return wrapped def __enter__(self): with self._cleanup_on_error(): self.wrappers = [self.enter_context(r) for r in self.resources] return self.wrappers # NB: ExitStack.__exit__ is already correct

Now your Foo() class is trivial:

import io class Foo(MultiResourceManager): def __init__(self, *paths): super().__init__(paths, io.FileIO, io.FileIO.close)

This is nice because we don't need any try-except blocks -- you're probably only using ContextManagers to get rid of those in the first place!

Then you can use it like you wanted to (note MultiResourceManager.__enter__ returns a list of objects given by the passed acquire_resource()):

if __name__ == '__main__': open('/tmp/a', 'w').close() open('/tmp/b', 'w').close() with Foo('/tmp/a', '/tmp/b') as (f1, f2): print('opened {0} and {1}'.format(f1.name, f2.name))

We can replace io.FileIO with debug_file as in the following snippet to see it in action:

 class debug_file(io.FileIO): def __enter__(self): print('{0}: enter'.format(self.name)) return super().__enter__() def __exit__(self, *exc_info): print('{0}: exit'.format(self.name)) return super().__exit__(*exc_info)

Then we see:

/tmp/a: enter /tmp/b: enter opened /tmp/a and /tmp/b /tmp/b: exit /tmp/a: exit

If we added import os; os.unlink('/tmp/b') just before the loop we'd see:

/tmp/a: enter /tmp/a: exit Traceback (most recent call last): File "t.py", line 58, in <module> with Foo('/tmp/a', '/tmp/b') as (f1, f2): File "t.py", line 46, in __enter__ self.wrappers = [self.enter_context(r) for r in self.resources] File "t.py", line 46, in <listcomp> self.wrappers = [self.enter_context(r) for r in self.resources] File "t.py", line 38, in enter_context wrapped = super().enter_context(self.acquire_resource(resource)) FileNotFoundError: [Errno 2] No such file or directory: '/tmp/b'

You can see /tmp/a is closed correctly.

Improve this answer

Comments

6

I think use a helper is better:

from contextlib import ExitStack, contextmanager class Foo: def __init__(self, i, o): self.i = i self.o = o @contextmanager def multiopen(i, o): with ExitStack() as stack: i = stack.enter_context(open(i)) o = stack.enter_context(open(o)) yield Foo(i, o)

The usage is close to native open:

with multiopen(i_name, o_name) as foo: pass
Improve this answer

8 Comments

Your first snippet exhibits the same bugs as the one I wrote after I tinkered about writing stuff such as (what if I get an exception while doing the second open?); also, why should I remember to close() stuff? It's just boilerplate code to write and to potentially get wrong; the point of context managers (and destructors in C++) is to do let this be handled automatically, regardless of how we are exiting the context.
I didn't considered that before. I'm thinking about it now. But your example using contextlib.ExitStack is still not a good idea as in this case, users have to use with statement or the instance they get has totally different behavior. For close() stuff, your point also makes sense. But "The Zen of Python" says "Explicit is better than implicit." And at least for me I will explicitly close underlying objects so that I know what I'm doing completely. Surely it's also OK to use __enter__/__exit__ pair.
That example can be easily adjusted by moving the whole initialization in __init__ instead of __enter__; in that case even using it "manually" (creation and explicit close()) works correctly (example).
As for the explicit is better than implicit, it all depends. Resources cleanup is one of those areas where the more automatic the better, as developers are notoriously bad at it, especially in a language with exceptions. It's not a chance that Python has automatic memory management, even if "explicit is better than explicit".
Yep it depends. And about your example, I write another example which I think is closer to native open and simpler.
|
4

Well, if you want to process certainly for file handlers, the simplest solution is just to pass filehandlers directly to your class instead of filenames.

with open(f1, 'r') as f1, open(f2, 'w') as f2: with MyClass(f1, f2) as my_obj: ...

If you don't need custom __exit__ functionality you may even skip nested with.

If you really want to pass filenames to __init__, your problem could be solved like this:

class MyClass: input, output = None, None def __init__(self, input, output): try: self.input = open(input, 'r') self.output = open(output, 'w') except BaseException as exc: self.__exit___(type(exc), exc, exc.__traceback__) raise def __enter__(self): return self def __exit__(self, *args): self.input and self.input.close() self.output and self.output.close() # My custom __exit__ code

So, it really depends on your task, python have plenty options to work with. In the end of the day - pythonic way is to keep your api simple.

Improve this answer

1 Comment

I think allocating resource in __init__ is an anti-pattern, explained in stackoverflow.com/questions/13074847/… .
0

Simplified solution using contextmanager

Sraw's answer can be simplified as follows:

from contextlib import contextmanager class Foo: def __init__(self, i, o): self.i = i self.o = o @contextmanager def multiopen(i, o): with open(i) as i, open(o) as o: yield Foo(i, o)

You don't need an explicit ExitStack in this case.

Of course, you can use it as follows:

with multiopen() as foo: print(foo)

If the number of objects is not fixed, then it's still recommended to use ExitStack as shown in Sraw's answer.

A context manager that is designed to make it easy to programmatically combine other context managers and cleanup functions, especially those that are optional or otherwise driven by input data.

If you need the resulting object to be both a contextmanager and a class

Use case described in https://stackoverflow.com/a/48957756/5267751 .

For example you may want something like the following to work:

@multiopen() def f(): ... with multiopen() as foo: ...

For this, the object being returned by multiopen() must implement all of __call__, __enter__ and __exit__.

The simplest solution is the following.

@contextmanager def _multiopen(i, o): with open(i) as i, open(o) as o: yield Foo(i, o) class multiopen(ContextDecorator): def __init__(self, i, o): self._multiopen_instance = _multiopen(i, o) def __enter__(self): self._multiopen_instance.__enter__() return self def __exit__(self, exc_type, exc_value, traceback): return self._multiopen_instance.__exit__(exc_type, exc_value, traceback)

That is, still make a contextmanager, but wrap over it with a ContextDecorator class.

Of course, this may looks ugly (you need the @contextmanager decorator, then immediately throw it away and make a class manually). Actually, it's not that bad --- even the recipe in the documentation https://docs.python.org/3/library/contextlib.html#cleaning-up-in-an-enter-implementation reduces a contextmanager.

If you refuse to use @contextmanager

You can also use pop_all --- the core idea is that

pop_all allows you to break up a with block.

This is exactly what you need to do in order to break the with block into the __enter__ method and the __exit__ method.

Like this: the code block

with ExitStack() as stack: stack.enter_context(A(0)) stack.enter_context(A(0)) BODY

can be broken down such that BODY no longer remains in the with block as follows:

with ExitStack() as stack: stack.enter_context(A(0)) stack.enter_context(A(0)) stack2 = stack.pop_all() # at this point stack.close() is called, but it does not call __exit__ on any A object. BODY stack2.__exit__(None, None, None)

Of course, you still need to remember to handle the case BODY raises an error, so:

with ExitStack() as stack: stack.enter_context(A(0)) stack.enter_context(A(0)) stack2 = stack.pop_all() with stack2: BODY

So, as a class:

class Multiopen: def __enter__(self): with ExitStack() as stack1: stack1.enter_context(...) stack1.enter_context(...) self._stack2=stack1.pop_all() def __exit__(self, *args): self._stack.close()

If any of the __enter__ raises an error, stack1 will unwind them. If none of them raises an error, self._stack2 will unwind them instead.

If you want to implement it yourself

The complexity is not just with ExitStack, it is also with the with statement itself.

From the documentation:

The following code:

with EXPRESSION as TARGET: SUITE

is semantically equivalent to:

manager = (EXPRESSION) enter = type(manager).__enter__ exit = type(manager).__exit__ value = enter(manager) hit_except = False try: TARGET = value SUITE except: hit_except = True if not exit(manager, *sys.exc_info()): raise finally: if not hit_except: exit(manager, None, None, None)

So, if we want to make a class, of course we cannot use with statement.

I can't figure out how to implement this correctly yet.

  • If the first object's __enter__ raises an exception, then we must not call the first object's __exit__.
  • However, if the second object's __enter__ raises an exception, we must call the first object's __exit__.
  • Furthermore, if the second object's __exit__ raises an exception, we must proceed to continue to call the first object's __exit__.
  • The first object's __exit__ may, in turn, suppress that exception.

(In this context, the open() should be seen as the __enter__. It may be more idiomatic to write a contextmanager that represents a file.)

Improve this answer

Comments

0

Here is my idea for composing context managers using ExitStack:

class MultiCM(ExitStack): def __init__(self, *cms): super().__init__() self.cms = cms def __enter__(self): with ExitStack() as stack: for cm in self.cms: stack.enter_context(cm) self.push(stack.pop_all()) return self

It follows the "Cleaning up in an __enter__ implementation" example from the documentation. (The main difficulty is the possibility or exceptions inside __enter__)

Here is a complete demonstration:

from contextlib import AbstractContextManager, ExitStack class CountedObjectsMixin: count = 0 @classmethod def _count(cls): cls.count += 1 return cls.count def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.number = self._count() def __str__(self): return f"[{ type(self).__name__ } { self.number }]" class CMDebugMixin(CountedObjectsMixin): def __enter__(self): print(f">.. Entering { self } ...") result = super().__enter__() print(f">>> Entered { self }") return result def __exit__(self, *exc): print(f"<.. Exiting { self } ...") result = super().__exit__(*exc) print(f"<<< Exited { self }") return result def debug_context_manager(cm_cls): return type(cm_cls.__name__, (CMDebugMixin, cm_cls), {}) @debug_context_manager class SimpleCM(AbstractContextManager): def __init__(self, broken=False): self.broken = broken def __enter__(self): if self.broken: print('Bang!') raise RuntimeError return self def __exit__(self, *exc): pass @debug_context_manager class MultiCM(ExitStack): def __init__(self, *cms): super().__init__() self.cms = cms def __enter__(self): with ExitStack() as stack: for cm in self.cms: stack.enter_context(cm) self.push(stack.pop_all()) return self if __name__ == '__main__': try: with MultiCM(SimpleCM(), SimpleCM()): print('Bang!') raise RuntimeError except RuntimeError: pass print("---") try: with MultiCM(SimpleCM(broken=True), SimpleCM()): pass except RuntimeError: pass print("---") try: with MultiCM(SimpleCM(), SimpleCM(broken=True)): pass except RuntimeError: pass

The output:

>.. Entering [MultiCM 1] ... >.. Entering [SimpleCM 1] ... >>> Entered [SimpleCM 1] >.. Entering [SimpleCM 2] ... >>> Entered [SimpleCM 2] >>> Entered [MultiCM 1] Bang! <.. Exiting [MultiCM 1] ... <.. Exiting [SimpleCM 2] ... <<< Exited [SimpleCM 2] <.. Exiting [SimpleCM 1] ... <<< Exited [SimpleCM 1] <<< Exited [MultiCM 1] --- >.. Entering [MultiCM 2] ... >.. Entering [SimpleCM 3] ... Bang! --- >.. Entering [MultiCM 3] ... >.. Entering [SimpleCM 5] ... >>> Entered [SimpleCM 5] >.. Entering [SimpleCM 6] ... Bang! <.. Exiting [SimpleCM 5] ... <<< Exited [SimpleCM 5]
Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.