embarrassingly parallel problem in python

Question

I have 634 *.npy files, each contain a 2D numpy array of shape (8194, 76). I want to use STL decomposition on each column five time with different frequencies. so what I'm trying to do is:

for file in files: for column in columns: for freq in frequencies: res = STL(file[:,column], period = freq) decomposed = np.vstack((res.trend, res.seasonal, res.resid)).T np.save(decompoesd)

finally the shape of decomposed should be (8194,1140). How can I parallize this? as it would take over 2 months to run in serial implementation.

Sorry for the late feedback @ArtiomKozyrev, It worked perfectly by parallelizing file level, thank you so much for your help. I made simulate_cpu_bound(file) takes a file process it and write it to a disk. I will try next parallelize the CPU heavy task itself, any hint on how to group the res output from each processor and write it to a disk using single process then only after that, all other processors take the next file and do the same — Tamer Adel
– Tamer Adel, Commented Dec 11, 2021 at 16:28

Artiom Kozyrev · Accepted Answer · 2021-12-08 15:30:51Z

1

You can do something like that:

from concurrent.futures import ProcessPoolExecutor FILES = ["a", "b", "c", "d", "e", "f", "g", "h"] def simulate_cpu_bound(file): 2 ** 100000000 # cpu heavy task # or just use time.sleep(n), where n - number of seconds return file if __name__ == '__main__': with ProcessPoolExecutor(8) as f: res = f.map(simulate_cpu_bound, FILES) res = list(res) print(res)

edited Dec 8, 2021 at 15:30

answered Dec 8, 2021 at 12:38

Artiom Kozyrev

3,9062 gold badges18 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Booboo Over a year ago

multithreading is definitely not what you want to use for a CPU heavy task unless that task is implemented, for example, as a CPU-language -implemented function that releases the Global Interpreter Lock (GIL). Did you try timing calling simulate_cpu_bound 8 times in a loop and then timing your multithreading version? You will not see much of a difference. Of course, if you change the function to just the sleep version you will because sleep is not CPU-intensive and does release the GIL.. There I would expect a time reduction of approximately 8-fold.

Artiom Kozyrev Over a year ago

@Booboo My mistake, I even did not pay attention that imported ThreadPoolExecutor instead of ProcessPoolExecutor, thanks. I made the required changes.

Collectives™ on Stack Overflow

embarrassingly parallel problem in python

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related