1

Currently I have a program that has proxies and makes a request to get my ip address with that proxy and returns it back in json.

An example request back is this:

Got Back: {'ip': '91.67.240.45', 'country': 'Germany', 'cc': 'DE'}

I want my program to try and make a request to the url and if it does not get the request because the proxy is down I want to try again 5 times before moving onto the next ip address.

I thought this except block would work but it is not breaking out of the loop when the 5 iterations are over and I am not sure why.

My program does however work when the proxy is up for the first try as it breaks after the first attempt and then moves onto the next ip address.

Here is what I currently have:

import requests import time proxies = [ "95.87.220.19:15600", "91.67.240.45:3128", "85.175.216.32:53281", "91.236.251.131:8118", "91.236.251.131:8118", "88.99.10.249:1080", ] def sol(ip): max_tries = 5 for i in range(1, max_tries+1): try: print(f"Using Proxy: {ip}") r = requests.get('https://api.myip.com', proxies={"https": ip}) print(f"Got Back: {r.json()}") break except OSError: time.sleep(5) print(f"Retrying...: {i}") break for i in proxies: sol(i)

How can I make it s my loop has 5 tries before moving onto the next ip address.

Improve this question
2
  • an easy way to do this is have a variable that starts at 0 and add 1 to it during each iteration. Instead of using a for loop, use a while loop. while variable <= 5. Commented Jul 12, 2020 at 1:43
  • The retrying module is really nice for this kind of use-case: pypi.org/project/retrying Commented Jul 12, 2020 at 1:44

3 Answers 3

Reset to default
1

My program does however work when the proxy is up for the first try as it breaks after the first attempt and then moves onto the next ip address.

It does this unconditionally, because you have an unconditional break after the except block. Code keeps going past a try/except when the except is entered, assuming it doesn't have an abnormal exit of its own (another exception, return etc.).

So,

I thought this except block would work but it is not breaking out of the loop when the 5 iterations are over and I am not sure why.

It doesn't break out "after the 5 iterations are over" because it breaks out after the first iteration, whether or not that attempt was successful.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

What can I do to make sure that the except will break out on the last iteration then?
It just will naturally. Why would you expect otherwise? The except block finishes running, then the for loop runs out of values from the range, then there is nothing left in the function. The only reason to have break in the try block is so that it doesn't do the rest of the 5 tries when the proxy is successful.
If the question is "how do I make the loop outside the function stop trying proxies when the function succeeds in finding a proxy", you need to return something to communicate that rather than simply letting the function reach the end. For example, you could return the request object that you get from the requests.get call (i.e. r); and then the loop will see whether a given proxy address found the a requests object or not (falling off the end of the function and returning None).
1

Using retrying would look something like the following:

from retrying import retry import requests @retry(stop_max_attempt_number=5) def bad_host(): print('trying get from bad host') return requests.get('https://bad.host.asdqwerqweo79ooo/') try: bad_host() except IOError as ex: print(f"Couldn't connect because: {str(ex)}")

...which give the following output:

trying get from bad host trying get from bad host trying get from bad host trying get from bad host trying get from bad host Couldn't connect to bad host because: HTTPSConnectionPool(host='bad.host.asdqwerqweo79ooo', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x10b6bd910>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

Getting fancy

If you want to get fancy, you could also add things like exponential backoff and selectively retrying certain exceptions.

Here's an example:

import random import time def retry_ioerror(exception): return isinstance(exception, IOError) @retry( wait_exponential_multiplier=100, wait_exponential_max=1000, retry_on_exception=retry_ioerror, stop_max_attempt_number=10) def do_something(): t = time.time() print(f'trying {t}') r = random.random() if r > 0.9: return 'yay!' if r > 0.8: raise RuntimeError('Boom!') else: raise IOError('Bang!') try: result = do_something() print(f'Success! {result}') except RuntimeError as ex: print(f"Failed: {str(ex)}")
Improve this answer

1 Comment

would you please be so kind and post small example of this --> exponential backoff and selectively retrying certain exceptions
0

If I understand correctly, you can just remove break from the last line. If you have an unconditional break in a loop, it will always iterate once.

def sol(ip): max_tries = 5 for i in range(1, max_tries+1): try: print(f"Using Proxy: {ip}") r = requests.get('https://api.myip.com', proxies={"https": ip}) print(f"Got Back: {r.json()}") break except OSError: time.sleep(5) print(f"Retrying...: {i}") # break <---- Remove this line
Improve this answer

Comments