I'm trying to create simple web monitoring script which sends GET request to urls in list periodically and asynchronously. Here is my request function:
def request(url,timeout=10): try: response = requests.get(url,timeout=timeout) response_time = response.elapsed.total_seconds() if response.status_code in (404,500): response.raise_for_status() html_response = response.text soup = BeautifulSoup(html_response,'lxml') # process page here logger.info("OK {}. Response time: {} seconds".format(url,response_time)) except requests.exceptions.ConnectionError: logger.error('Connection error. {} is down. Response time: {} seconds'.format(url,response_time)) except requests.exceptions.Timeout: logger.error('Timeout. {} not responding. Response time: {} seconds'.format(url,response_time)) except requests.exceptions.HTTPError: logger.error('HTTP Error. {} returned status code {}. Response time: {} seconds'.format(url,response.status_code, response_time)) except requests.exceptions.TooManyRedirects: logger.error('Too many redirects for {}. Response time: {} seconds'.format(url,response_time)) except: logger.error('Content requirement not found for {}. Response time: {} seconds'.format(url,response_time)) And here where I call this function for all urls:
def async_requests(delay,urls): for url in urls: async_task = make_async(request,delay,url,10) loop.call_soon(delay,async_task) try: loop.run_forever() finally: loop.close() delay argument is interval for loop which describes how often function needs to be executed. In order to loop request I created something like this:
def make_async(func,delay,*args,**kwargs): def wrapper(*args, **kwargs): func(*args, **kwargs) loop.call_soon(delay, wrapper) return wrapper every time I execute async_requests I get this error for each url:
Exception in callback 1.0(<function mak...x7f1d48dd1730>) handle: <Handle 1.0(<function mak...x7f1d48dd1730>)> Traceback (most recent call last): File "/usr/lib/python3.5/asyncio/events.py", line 125, in _run self._callback(*self._args) TypeError: 'float' object is not callable Also request functions for each urls are not being executed periodically as intended. Also my print function which goes after async_requests is not executed either:
async_requests(args.delay,urls) print("Starting...") I understand that I'm doing something wrong in code but I can't figure how to solve this problem. I'm beginner in python and not very experienced with asyncio. Summarizing what I want to achive:
- Run asynchronously and periodcally
requestfor particular url without blocking main thread. - Run
async_requestsasynchronously so I could launch a simple http server for example in same thread.
call_soondoesn’t work the way you’re using it. It takes a callback as its first argument. Everything that follows is an argument passed to the callback. The callback is called as soon as control returns back to the event loop. What you’re looking for iscall_later.