16

What are use cases in python 3 of writing a custom __del__ method or relying on one from stdlib1? That is, in what scenario is it reasonably safe, and can do something that's hard to do without it?

For many good reasons (1 2 3 4 5 6), the usual recommendation is to avoid __del__ and instead use context managers or perform the cleanup manually:

  1. __del__ is not guaranteed to be called if objects are alive on intrepreter exit2.
  2. At the point one expects the object can be destroyed, the ref count may actually be non-zero (e.g., a reference may survive through a traceback frame held onto by a calling function). This makes the destruction time far more uncertain than the mere unpredictability of gc implies.
  3. Garbage collector cannot get rid of cycles if they include more than 1 object with __del__
  4. The code inside __del__ must be written super carefully:
    • object attributes set in __init__ may not be present since __init__ might have raised an exception;
    • exceptions are ignored (only printed to stderr);
    • globals may no longer be available.

Update:

PEP 442 has made significant improvements in the behavior of __del__. It seems though that my points 1-4 are still valid?

Update 2:

Some of the top python libraries embrace the use of __del__ in the post-PEP 442 python (i.e., python 3.4+). I guess my point 3 is no longer valid after PEP 442, and the other points are accepted as unavoidable complexity of object finalization.

1I expanded the question from just writing a custom __del__ method to include relying on __del__ from stdlib.

2It seems that __del__ is always called on interpreter exit in the more recent versions of Cpython (does anyone have a counter-example?). However, it doesn't matter for the purpose of __del__'s usablity: the docs explicitly provide no guarantee about this behavior, so one cannot rely on it (it may change in future versions, and it may be different in non-CPython interpreters).

Improve this question
7
  • 1
    Why do you exclude the standard library in this question? Commented Apr 27, 2017 at 19:57
  • 1
    Because I (over)used __del__ in my code, and I am planning to completely abandon it. Before I do, I want to make sure I'm not missing situations where I shouldn't avoid it. Since I (and most people on SO) don't contribute to stdlib, I wanted to keep the question simpler by excluding it. Commented Apr 27, 2017 at 19:59
  • @ali_m I was thinking about it too. So let's say your objects' lifetime is too open-ended to enclose in a context manager, so you end up relying on __del__. You must care about cleanup enough to bother writing __del__. And yet you don't mind if the cleanup occsionally never happens or happens much later than expected (since that's the semantics of __del__). What kind of cleanup would be a good fit? Maybe freeing memory, but that's already done by gc. Maybe helping gc to free memory where it struggles? Commented Apr 27, 2017 at 22:39
  • @max In some cases you only require the __del__ call if you're close to running out of memory and most python implementations collect objects when memory is getting "sparse". So you don't care about "immediate cleanup" but only about "eventual cleanup". Commented Apr 27, 2017 at 22:44
  • @max gc can't free memory that was allocated outside of Python (e.g. by a call to malloc from some shared library). Commented Apr 27, 2017 at 22:45

3 Answers 3

Reset to default
10

Context managers (and try/finally blocks) are somewhat more restrictive than __del__. In general they require you to structure your code in such a way that the lifetime of the resource you need to free doesn't extend beyond a single function call at some level in the call stack, rather than, say, binding it to the lifetime of a class instance that could be destroyed at unpredictable times and places. It's usually a good thing to restrict the lifetime of resources to one scope, but there sometimes edge cases where this pattern is an awkward fit.

The only case where I've used __del__ (aside from for debugging, c.f. @MSeifert's answer) is for freeing memory allocated outside of Python by an external library. Because of the design of the library I was wrapping, it was difficult to avoid having a large number of objects that held pointers to heap-allocated memory. Using a __del__ method to free the pointers was the easiest way to do cleanup, since it would have been impractical to enclose the lifespan of each instance inside a context manager.

Improve this answer
Sign up to request clarification or add additional context in comments.

7 Comments

What did your __del__ do that the normal garbage collection couldn't?
@max It freed memory that had been allocated outside of Python (by a C function in the library I was wrapping), and was therefore invisible to gc.
I'll accept your answer even though both are very helplful because I think freeing of malloc-allocated memory is a perfect production use case. From my question, point (1) is irrelevant since the process memory will be freed on exit; point (2) is less relevant because (as @MSeifert points out) as long as gc frees it before memory completely runs out, it's good enough; (3) and (4) are just points that implementation needs to be careful about.
Wouldn't weakref callbacks be suitable for this purpose, without the downsides of __del__?
@MariusGedminas I asked a question about this, since I think it's too much to add more side questions to this post.
|
8

One use-case is debugging. If you want to track the lifetime of a specific object it's convenient to write a temporary __del__ method. It can be used to do some logging or just to print something. I have used it a few times especially when I was interested in when and how instances are deleted. It's sometimes good to know when you create and discard a lot of temporary instances. But as I said I only ever used this to satisfy my curiosity or when debugging.

Another use-case is subclassing a class that defines a __del__ method. Sometimes you find a class that you want to subclass but the internals require you to actually override __del__ to control the order in which the instance is cleaned up. That's very rare too because you need to find a class with __del__, you nee to subclass it and you need to introduced some internals that actually require to call the superclass __del__ at exactly the right time. I actually did that once but I don't remember where and why it was important (maybe I didn't even know about alternatives then, so treat this as possible use-case).

When you wrap an external object (for example a object that isn't tracked by Python) that really, really needs to be deallocated even if someone "forgets" (I suspect a lot of people just omit them on purpose!) to use the context manager that you provided.

However all these cases are (or should be) very, very rare. Actually it's a bit like with metaclasses: They are fun and it's really cool to understand the concepts because you can probe the "fun parts" of python. But in practice:

If you wonder whether you need them [metaclasses], you don’t (the people who actually need them know with certainty that they need them, and don’t need an explanation about why).

Citation (probably) from Tim Peters (I haven't found the original reference).

Improve this answer

5 Comments

The main problem that prevents destructors from being useful in production code seems to be the lack of predictability of when or whether they are called. The metaclass, being slightly complex but perfectly predictable, is a lot more useful!
@max Yes, I guess the comparison isn't that useful after all. I was purely thinking about use-cases: Both have use-cases, but they tend to be really sparse (and really complex).
BTW, another technique (which I just found out about) to help track the objects' lifetime, is to set PYTHONVERBOSE=2.
@max Isn't that just about modules? I never actually used it yet, I'll definetly give it a try :) But in my debugging scenarios I was only interested in one or two classes so there's no need to track all. :)
Ah yes, you're right, it only includes module cleanup info.
1

One case where i always use __del__, is for closing a aiohttp.ClientSession object.

When you don't, aiohttp will print warnings about the unclosed client session.

Improve this answer

1 Comment

But wouldn't that be unreliable, since based on the python documentation, your __del__ may never be called?

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.