Multiprocessing file reading and output

Question

I am trying to process files in over 1000 directories. However, as the size of the directory is very big, i have to run the process concurrently to save time. The dir [:100] is so that that function has only 100 directories to process.However, the 2nd Thread does not start until the first Thread has finished.

def sort(start,end): dir = os.list(filepath) dir = dir [start:end] for file in dir: ~process~ print result if __name__ == '__main__': Thread(target = Sort(0,100)).start() Thread(target = Sort(100,200)).start()

I have tried duplicating the Sort function as Sort2(): but this yields the same result

Thread(target = Sort(0,100)).start() Thread(target = Sort2(100,200)).start()

Any help or direction is greatly apprenticed

possible duplicate of python threading blocks

Gustavo Meira
– Gustavo Meira

2014-08-08 09:04:20 +00:00
Commented Aug 8, 2014 at 9:04 — Gustavo Meira
– Gustavo Meira, Commented Aug 8, 2014 at 9:04

Community · Accepted Answer · 2017-05-23 12:13:23Z

The problem is that Sort and Sort2 are being evaluated before you start your threads. When you say Sort(...), the () is an operator that executes the Sort function.

This answer could say more about your problem.

You'll have to use the args parameter for the Thread constructor. For example:

Thread(target=Sort, args=(0,100)).start()

Ketouem · Accepted Answer · 2014-08-08 09:13:13Z

import multiprocessing def worker_sort(): # includes the code of your sorting function pass if __name__ == '__main__': jobs = [] for i in range(2): p = multiprocessing.Process(target=worker_sort, args=(0,100)) jobs.append(p) p.start()

I'd recommend some reading about the multiprocessing module in the official documentation

Collectives™ on Stack Overflow

Multiprocessing file reading and output

2 Answers 2

1 Comment

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Linked

Related