Using threads and processes together with shared queues in Python

Question

I'm trying to use a multithreading queue and a multiprocessing queue at the same time. The threading queue will be used by 20 threads to retrieve many web pages. I then want to put the pages into a multiprocess queue so that 4 process workers can crunch the data. Below is my basic structure. My issue is that, the work queue, gives an error saying Queue is not iterable. I think the multithreading queue is overwriting the multiprocess queue but I really don't know what's wrong.

ticker_queue = Queue() work_queue = Queue() tickers = get_tickers() for i in tickers: ticker_queue.put(i) for i in range(20): t = Thread(target=network_worker, args = (ticker_queue, work_queue)).start() for i in range(4): p = Process(target = worker, args = (work_queue)).start()

Here is the traceback

Traceback (most recent call last): File "OneDrive\Python\andys.py", line 108, in <module> p = Process(target = worker, args = (work_queue)).start() File "C:\Python27\lib\multiprocessing\process.py", line 104, in __init__ self._args = tuple(args) TypeError: 'Queue' object is not iterable

boil your problem down to a simple self-contained example, then post your example and the traceback. Nobody's seen professor Xavier for a while.. ;-) — thebjorn
– thebjorn, Commented Jan 24, 2015 at 18:31
I have to go to work in a few, I will post a self-contained example later tonight if no one has responded. Thanks. — nicholas.reichel
– nicholas.reichel, Commented Jan 24, 2015 at 18:35
You're missing a comma: p = Process(target = worker, args = (work_queue,)).start() — thebjorn
– thebjorn, Commented Jan 24, 2015 at 18:36
WOW! A COMMA! Thanks a bunch! It worked. Programming syntax sometimes... -.- — nicholas.reichel
– nicholas.reichel, Commented Jan 24, 2015 at 18:40

thebjorn · Accepted Answer · 2015-01-24 18:42:33Z

There is a comma missing in

p = Process(target = worker, args = (work_queue)).start()

since

(work_queue)

is just an expression, while what you want is a 1-element tuple:

(work_queue,)

notice the additional comma.

Collectives™ on Stack Overflow

Using threads and processes together with shared queues in Python

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related