26

I have a bit of multiprocessing Python code that looks a bit like this:

import time from multiprocessing import Pool import numpy as np class MyClass(object): def __init__(self): self.myAttribute = np.zeros(100000000) # basically a big memory struct def my_multithreaded_analysis(self): arg_lists = [(self, i) for i in range(10)] pool = Pool(processes=10) result = pool.map(call_method, arg_lists) print result def analyze(self, i): time.sleep(10) return i ** 2 def call_method(args): my_instance, i = args return my_instance.analyze(i) if __name__ == '__main__': my_instance = MyClass() my_instance.my_multithreaded_analysis()

After reading answers about how memory works in other StackOverflow answers such as this one Python multiprocessing memory usage I was under the impression that this would not use memory in proportion to how many processes I used for multiprocessing, since it is copy-on-write and I have not modified any of the attributes of my_instance. However, I do see high memory for all processes when I run top it says most of my processes are using a lot of memory (this is top output from OSX, but I can replicate on Linux).

My question is basically, am I interpreting this correctly in that my instance of MyClass is actually duplicated across the pool? And if so, how can I prevent this; should I just not use a construction like this? My goal is to reduce memory usage for a computational analysis. 

PID COMMAND %CPU TIME #TH #WQ #PORT MEM PURG CMPRS PGRP PPID STATE 2494 Python 0.0 00:01.75 1 0 7 765M 0B 0B 2484 2484 sleeping 2493 Python 0.0 00:01.85 1 0 7 765M 0B 0B 2484 2484 sleeping 2492 Python 0.0 00:01.86 1 0 7 765M 0B 0B 2484 2484 sleeping 2491 Python 0.0 00:01.83 1 0 7 765M 0B 0B 2484 2484 sleeping 2490 Python 0.0 00:01.87 1 0 7 765M 0B 0B 2484 2484 sleeping 2489 Python 0.0 00:01.79 1 0 7 167M 0B 597M 2484 2484 sleeping 2488 Python 0.0 00:01.77 1 0 7 10M 0B 755M 2484 2484 sleeping 2487 Python 0.0 00:01.75 1 0 7 8724K 0B 756M 2484 2484 sleeping 2486 Python 0.0 00:01.78 1 0 7 9968K 0B 755M 2484 2484 sleeping 2485 Python 0.0 00:01.74 1 0 7 171M 0B 594M 2484 2484 sleeping 2484 Python 0.1 00:16.43 4 0 18 775M 0B 12K 2484 2235 sleeping
Improve this question
1
  • 1
    How did you generate this profiler result? Commented Mar 22, 2021 at 17:06

2 Answers 2

Reset to default
53
+50

Anything sent to pool.map (and related methods) isn't actually using shared copy-on-write resources. The values are "pickled" (Python's serialization mechanism), sent over pipes to the worker processes and unpickled there, which reconstructs the object in the child from scratch. Thus, each child in this case ends up with a copy-on-write version of the original data (which it never uses, because it was told to use the copy sent via IPC), and a personal recreation of the original data that was reconstructed in the child and is not shared.

If you want to take advantage of forking's copy-on-write benefits, you can't send data (or objects referencing the data) over the pipe. You have to store them in a location that can be found from the child by accessing their own globals. So for example:

import os import time from multiprocessing import Pool import numpy as np class MyClass(object): def __init__(self): self.myAttribute = os.urandom(1024*1024*1024) # basically a big memory struct(~1GB size) def my_multithreaded_analysis(self): arg_lists = list(range(10)) # Don't pass self pool = Pool(processes=10) result = pool.map(call_method, arg_lists) print result def analyze(self, i): time.sleep(10) return i ** 2 def call_method(i): # Implicitly use global copy of my_instance, not one passed as an argument return my_instance.analyze(i) # Constructed globally and unconditionally, so the instance exists # prior to forking in commonly accessible location my_instance = MyClass() if __name__ == '__main__': my_instance.my_multithreaded_analysis()

By not passing self, you avoid making copies, and just use the single global object that was copy-on-write mapped into the child. If you needed more than one object, you might make a global list or dict mapping to instances of the object prior to creating the pool, then pass the index or key that can look up the object as part of the argument(s) to pool.map. The worker function then uses the index/key (which had to be pickled and sent to the child over IPC) to look up the value (copy-on-write mapped) in the global dict (also copy-on-write mapped), so you copy cheap information to lookup expensive data in the child without copying it.

If the objects are smallish, they'll end up copied even if you don't write to them. CPython is reference counted, and the reference count appears in the common object header and is updated constantly, just by referring to the object, even if it's a logically non-mutating reference. So small objects (and all the other objects allocated in the same page of memory) will be written, and therefore copied. For large objects (your hundred million element numpy array), most of it would remain shared as long as you didn't write to it, since the header only occupies one of many pages

Changed in python version 3.8: On macOS, the spawn start method is now the default. See mulitprocessing doc. Spawn is not leveraging copy-on-write.

Improve this answer
Sign up to request clarification or add additional context in comments.

15 Comments

Also note: If the objects are smallish, they'll end up copied even if you don't write to them. CPython is reference counted, and the reference count appears in the common object header and is updated constantly, just by referring to the object, even if it's a logically non-mutating reference. So small objects (and all the other objects allocated in the same page of memory) will be written, and therefore copied. For large objects (your hundred million element numpy array), most of it would remain shared as long as you didn't write to it, since the header only occupies one of many pages.
I've incorporated your comment into the answer body. The implication of that statement is that for vanilla Python data structures (lists, dicts etc), a copy is triggered at point of reference in the child process therefore you might as well pass the structure explicitly as a method parameter and be done with it. Would you know if a way exists to prevent this behaviour?
@iruvar: It's still cheaper to have it duplicated via COW than to pickle it, send it via a pipe, then unpickle it on the other side. And any stuff that isn't actually referenced (data created in the parent and not loaded in the workers) won't be duplicated. The only ways to "prevent" this behavior are to use non-CPython interpreters (though their GC process is likely to trigger similar behaviors), or use non-fork start methods (so you'll have to send stuff via pickling, but at least you've got far less that could potentially be copied).
@dre-hh: macOS defaults to using the 'spawn' method instead of 'fork' starting in 3.8, because macOS system frameworks are not fork-safe. The way 'spawn' works is very different from the way 'fork' works (it does a bunch of stuff to simulate forking, sort of, but COW isn't involved, at all). You can always try opting in to the 'fork' start method (at the expense of possibly crashing your code if you get unlucky on the fork timing).
@kwsp: In pool-of-workers scenario, most of the objects in the process will not be directly referenced by the tasks dispatched to the pool. They work with arguments passed to the worker, and perhaps a handful of globals. Cyclic GC runs eventually touch every (non-frozen, eventually, non-immortal) object that can potentially participate in a cycle, generally a few orders of magnitude more objects than what gets touched directly by the tasks dispatched to the worker. If you use a given object at all, yep, copied due to header writes, but you usually don't use most objects.
|
7

Alternatively, to take advantage of forking's copy-on-write benefits, while preserving some semblance of encapsulation, you could leverage class-attributes and @classmethods over pure globals. 

import time from multiprocessing import Pool import numpy as np class MyClass(object): myAttribute = np.zeros(100000000) # basically a big memory struct # myAttribute is a class-attribute @classmethod def my_multithreaded_analysis(cls): arg_list = [i for i in range(10)] pool = Pool(processes=10) result = pool.map(analyze, arg_list) print result @classmethod def analyze(cls, i): time.sleep(10) # If you wanted, you could access cls.myAttribute w/o worry here. return i ** 2 """ We don't need this proxy step ! def call_method(args): my_instance, i = args return my_instance.analyze(i) """ if __name__ == '__main__': my_instance = MyClass() # Note that now you can instantiate MyClass anywhere in your app, # While still taking advantage of copy-on-write forking my_instance.my_multithreaded_analysis()

Note 1: Yes, I admit that class-attributes and class-methods are glorified globals. But it buys a bit of encapsulation...

Note 2: Rather than explicitly creating your arg_lists above, you can implicitly pass the instance (self) to each task created by Pool, by passing the bound-instance method analyze(self) to Pool.map(), and shoot yourself in the foot even easier!

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.