Error: Time out while performing API call in Python

Question

I have a list (lst) which is a list of list. There are 19 elements in this list and each element has ~2500 strings.

lst [['A', 'B', 'C',...]['E', 'F', 'G',....][.....]]

I am using these strings (A,B....) to call an API endpoint ('q':element). However after ~1800 strings, I am getting a time out.

I am running following lines of code.

def get_val(element): url = 'https://www.xxxx/yyy/api/search' headers = {'Content-Type': 'application/json'} param = {'q': element, 'page' : 500} try: response = requests.get(url, headers = headers, params = param, timeout=(3.05, 27)) docs = response.json()['response']['docs'] for result in docs: file.write("%s\t%s\n" % (element,result['short_form'])) except Timeout: print('Timeout has been raised.') #loop through elements of list for i in lst: for element in i: get_val(element)

How can I modify my code to avoid this time out?

Sounds like the timeout is deliberately caused by a server-side mechanism to protect the server against automated mass requests. You may want to add significant delays between calls. — blhsing
– blhsing, Commented Jan 5, 2023 at 8:28
The timeout has nothing to do with your code. It just means that the API is taking too long to respond — jackal
– jackal, Commented Jan 5, 2023 at 8:34
I know it has already been explained (it's very likely a server side protection mechanism) but I just want to add that you can debug this kind of issue by using curl or some similar HTTP client to prove it's not an issue with your code. — A. Darwin
– A. Darwin, Commented Jan 5, 2023 at 8:38

buec95 · Accepted Answer · 2023-01-05 08:32:58Z

One reason for this timeout could be a protection against mass requests, that means, that there are too many requests in a short time.

To overcome this problem a short pause could be added after for example every 100 requests. However this is a try and error approach but it could work. Worst case would be to add a delay after every request.

import time time.sleep(0.5)

The parameter is added in seconds so 0.5 sec for example.

Collectives™ on Stack Overflow

Error: Time out while performing API call in Python

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related