Gevent incremental process through non blocking joinall()

Question

Here I want to make some modificatins for my setting.

I want response from multiple API calls within a single request made to my server. from all these API calls I want to combine results and return them as a response. Until here pretty much everything follows as given in examples of gevent documentation and over here. Now the catch here is that I want to pass response in incremental way, so if first API call has returned the result I will return this result to frontend in one long waited request and then wait for other API calls and pass them in same request to frontend.

I have tried to do this through code but I dont know how to proceed with this setting. The gevent .joinall() and .join() block untill all the greenlets are finished getting responses.

Any way I can procceed with gevent in this setting ?

Code I am using here is given on link https://bitbucket.org/denis/gevent/src/tip/examples/concurrent_download.py . Here the .joinall() in the last statement waits until all urls have complete giving responses, I want it to be non blocking so that I can process the responses in the callback function print_head() and return them incrementally.

#!/usr/bin/python # Copyright (c) 2009 Denis Bilenko. See LICENSE for details. """Spawn multiple workers and wait for them to complete""" urls = ['http://www.google.com', 'http://www.yandex.ru', 'http://www.python.org'] import gevent from gevent import monkey # patches stdlib (including socket and ssl modules) to cooperate with other greenlets monkey.patch_all() import urllib2 def print_head(url): print ('Starting %s' % url) data = urllib2.urlopen(url).read() print ('%s: %s bytes: %r' % (url, len(data), data[:50])) jobs = [gevent.spawn(print_head, url) for url in urls] gevent.joinall(jobs)

Denis · Accepted Answer · 2011-11-09 10:43:51Z

If you want to collect results from multiple greenlets, then modify print_head() to return the result and then use .get() method to collect them all.

Put this after joinall():

total_result = [x.get() for x in jobs]

Actually, joinall() is not even necessary in this case.

If print_head() looks like this:

def print_head(url): print ('Starting %s' % url) return urllib2.urlopen(url).read()

Then total_result will be a list of size 3 containing the responses from all the requests.

I tried modifying print_head() to return result and doing a .get() for each job in job list but the problem still remains here that .get() blocks, Can I use something non blocking instead and in the same execution flow keep monitoring for the result from each job.
.get() only blocks the current greenlet; the other jobs are running during get(); there's also 'value' property which is None which will be set to the return value of 'print_head' once it's done. Until the job is done, 'value' is None.

Collectives™ on Stack Overflow

Gevent incremental process through non blocking joinall()

1 Answer 1

2 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Related