I have the following code in Python:
from selenium.webdriver import Firefox from contextlib import closing with closing(Firefox()) as browser: browser.get(url) I would like to print the user-agent HTTP header and possibly change it. Is it possible?
There is no way in Selenium to read the request or response headers. You could do it by instructing your browser to connect through a proxy that records this kind of information.
The usual way to change the user agent for Firefox is to set the variable "general.useragent.override" in your Firefox profile. Note that this is independent from Selenium.
You can direct Selenium to use a profile different from the default one, like this:
from selenium import webdriver profile = webdriver.FirefoxProfile() profile.set_preference("general.useragent.override", "whatever you want") driver = webdriver.Firefox(profile) With Chrome, what you want to do is use the user-agent command line option. Again, this is not a Selenium thing. You can invoke Chrome at the command line with chrome --user-agent=foo to set the agent to the value foo.
With Selenium you set it like this:
from selenium import webdriver from selenium.webdriver.chrome.options import Options opts = Options() opts.add_argument("user-agent=whatever you want") driver = webdriver.Chrome(chrome_options=opts) Both methods above were tested and found to work. I don't know about other browsers.
Selenium does not have methods to query the user agent from an instance of WebDriver. Even in the case of Firefox, you cannot discover the default user agent by checking what general.useragent.override would be if not set to a custom value. (This setting does not exist before it is set to some value.)
Once the browser is started, however, you can get the user agent by executing:
agent = driver.execute_script("return navigator.userAgent") The agent variable will contain the user agent.
from selenium.webdriver import Firefox. I'm trying to figure out, how to set the profile for Firefox, I imported from selenium.webdriver.
User-Agent absolutely is a request header. See section 14.43: "The User-Agent request-header field contains information about the user agent originating the request." (Emphasis mine.)This is a short solution to change the request UserAgent on the fly.
Change UserAgent of a request with Chrome
from selenium import webdriver from selenium.webdriver.common.desired_capabilities import DesiredCapabilities driver = webdriver.Chrome(driver_path) driver.execute_cdp_cmd('Network.setUserAgentOverride', {"userAgent":"python 2.7", "platform":"Windows"}) driver.get('http://amiunique.org') then return your useragent:
agent = driver.execute_script("return navigator.userAgent") Some sources
The source code of webdriver.py from SeleniumHQ (https://github.com/SeleniumHQ/selenium/blob/11c25d75bd7ed22e6172d6a2a795a1d195fb0875/py/selenium/webdriver/chrome/webdriver.py) extends its functionalities through the Chrome Devtools Protocol
def execute_cdp_cmd(self, cmd, cmd_args): """ Execute Chrome Devtools Protocol command and get returned result We can use the Chrome Devtools Protocol Viewer to list more extended functionalities (https://chromedevtools.github.io/devtools-protocol/tot/Network#method-setUserAgentOverride) as well as the parameters type to use.
To build on Louis's helpful answer...
from selenium import webdriver from selenium.webdriver.common.desired_capabilities import DesiredCapabilities ... caps = DesiredCapabilities.PHANTOMJS caps["phantomjs.page.settings.userAgent"] = "whatever you want" driver = webdriver.PhantomJS(desired_capabilities=caps) The only minor issue is that, unlike for Firefox and Chrome, this does not return your custom setting:
driver.execute_script("return navigator.userAgent") So, if anyone figures out how to do that in PhantomJS, please edit my answer or add a comment below! Cheers.
Firefox Profile is deprecated, you have to use it in Firefox options like this:
opts = FirefoxOptions() opts.add_argument("--headless") opts.add_argument("--width=800") opts.add_argument("--height=600") opts.set_preference("general.useragent.override", "userAgent=Mozilla/5.0 (iPhone; CPU iPhone OS 15_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/101.0.4951.44 Mobile/15E148 Safari/604.1") To build on JJC's helpful answer that builds on Louis's helpful answer...
With PhantomJS 2.1.1-windows this line works:
driver.execute_script("return navigator.userAgent") If it doesn't work, you can still get the user agent via the log (to build on Mma's answer):
from selenium import webdriver import json from fake_useragent import UserAgent dcap = dict(DesiredCapabilities.PHANTOMJS) dcap["phantomjs.page.settings.userAgent"] = (UserAgent().random) driver = webdriver.PhantomJS(executable_path=r"your_path", desired_capabilities=dcap) har = json.loads(driver.get_log('har')[0]['message']) # get the log print('user agent: ', har['log']['entries'][0]['request']['headers'][1]['value']) 