1

I'm working on a project where i need to scrap some data using selenium.. For scrapping, i want to rotate my server IPs (my server has multiple IPs attached to single machine). For each request, i want to use a different IP so i don't get blocked. Not sure how this could be done. I'm using Firefox on ubuntu 16.04.

So far, by googling for many hours, i have found this but its not working. 

profile = webdriver.FirefoxProfile() profile.set_preference("network.proxy.type", 1) profile.set_preference("network.proxy.http", 'my-ip') profile.set_preference("network.proxy.http_port", 80) profile.set_preference("network.proxy.https", 'my-ip') profile.set_preference("network.proxy.https_port", 80) profile.update_preferences() driver = webdriver.Firefox(firefox_profile=profile) driver.get('https://api.ipify.org/') print driver.page_source

I'm testing with https://api.ipify.org/ just to find which IP is used for the generated request but it always shows me the primary IP of my server even though i have set different IP using the above code.

Any help/guidance or workaround would be highly appreciated.

Improve this question

1 Answer 1

Reset to default
1

What you would need is to bind Firefox to a different network interface, which is not possible.

A workaround would be to set up a proxy like squid on your server, configure it to route outgoing traffic through each interface based on a parameter you can control from selenium configuration for Firefox, for example the proxy port (it should be possible with squid acl).

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thanks, I'll try using squid

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.