timeout not respected in request.get()

Question

I can't understand why with this code the requests.get() returns immediately error and the 10 second timeout is not respected. If I remove the header parameter, so only requests.get(url, proxies={"http": proxy, "https": proxy}, verify=False,timeout=10), the timeout is respected and a request is returned successfully or failure if the timeout occurs. I need to insert headers and a timeout too to test proxies, how to do it?

from fake_useragent import UserAgent import requests ua = UserAgent() #get a list of proxy for i in range(1, len(proxies)): # Get a proxy from the pool proxy = next(proxy_pool) print("Request #%d" % i) try: response = requests.get(url, headers=ua.random(), proxies={"http": proxy, "https": proxy}, verify=False,timeout=10) break except: print("Skipping. Connnection error")

and if you replace the header by headers['User-Agent'] = "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1312.27 Safari/537.17" ??? — GiovaniSalazar
– GiovaniSalazar, Commented Dec 10, 2019 at 21:32
yes, it works, so now i think ua.random() generates bad user-agents. Do you know any better packages? Or a list of reliable user agents? @GiovaniSalazar — JayJona
– JayJona, Commented Dec 10, 2019 at 21:40

GiovaniSalazar · Accepted Answer · 2019-12-10 21:40:50Z

try with this

ua = UserAgent() header = {'User-Agent':str(ua.random)} response = requests.get(url, headers=header, verify=False,timeout=10)

Collectives™ on Stack Overflow

timeout not respected in request.get()

1 Answer 1

Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Related