I'm trying to parse a webpage using Java with URLConnection. I try to set up the user-agent like this:
java.net.URLConnection c = url.openConnection(); c.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2"); But the resulting user agent is the one I specify, with "Java/1.5.0_19" appended to the end. Is there a way to truly set the user agent without this addition?
Just for clarification: setRequestProperty("User-Agent", "Mozilla ...") now works just fine and doesn't append java/xx at the end! At least with Java 1.6.30 and newer.
I listened on my machine with netcat(a port listener):
$ nc -l -p 8080 It simply listens on the port, so you see anything which gets requested, like raw http-headers.
And got the following http-headers without setRequestProperty:
GET /foobar HTTP/1.1 User-Agent: Java/1.6.0_30 Host: localhost:8080 Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 Connection: keep-alive And WITH setRequestProperty:
GET /foobar HTTP/1.1 User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2 Host: localhost:8080 Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2 Connection: keep-alive As you can see the user agent was properly set.
Full example:
import java.io.IOException; import java.net.URL; import java.net.URLConnection; public class TestUrlOpener { public static void main(String[] args) throws IOException { URL url = new URL("http://localhost:8080/foobar"); URLConnection hc = url.openConnection(); hc.setRequestProperty("User-Agent", "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.2.2) Gecko/20100316 Firefox/3.6.2"); System.out.println(hc.getContentType()); } } 
Off hand, setting the http.agent system property to "" might do the trick (I don't have the code in front of me).
You might get away with:
System.setProperty("http.agent", ""); but that might require a race between you and initialisation of the URL protocol handler, if it caches the value at startup (actually, I don't think it does).
The property can also be set through JNLP files (available to applets from 6u10) and on the command line:
-Dhttp.agent= Or for wrapper commands:
-J-Dhttp.agent= 
its work for me set the User-Agent in the addRequestProperty.
URL url = new URL(<URL>); HttpURLConnection httpConn = (HttpURLConnection) url.openConnection(); httpConn.addRequestProperty("User-Agent","Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:56.0) Gecko/20100101 Firefox/56.0"); 
HttpURLConnection. You would have been right if he used URLConnection. Anyway, if you try to compile this line HttpURLConnection c = new URL("http://www.google.com").openConnection(); for example, javac will give you error: incompatible types on that line. I'm using Java 1.8.0 r172, so the most recent release of Java 8HttpURLConnection c = new URL("http://www.google.com").openConnection(); certainly requires a cast to HttpURLConnection - in Java 8 and also previous versions of Java.HTTP Servers tend to reject old browsers and systems.
The page Tech Blog (wh): Most Common User Agents reflects the user-agent property of your current browser in section "Your user agent is:", which can be applied to set the request property "User-Agent" of a java.net.URLConnection or the system property "http.agent".
