2

I'm trying to connect to a web page through a HttpURLConnection but it's not responding the same as a regular browser (firefox, chrome). I'm getting an error 500.

With the same code I can get "anyother" page (google, for example). My code is posted below, but I'm pretty sure it is ok.

Using "Live HTTP Headers" firefox addon I sent the same request and received a valid response.

//... String urlConsulta = "myURL"; URL url = new URL(urlConsulta); HttpURLConnection uConsulta = (HttpURLConnection)consulta.openConnection(); uConsulta.setDoOutput(true); uConsulta.setRequestMethod("POST"); uConsulta.addRequestProperty("Content-Type", "application/x-www-form-urlencoded"); uConsulta.addRequestProperty("Host", "host"); //...
Improve this question
2
  • 1
    @user470610: I've crawled webpages using Java and there is one thing I can tell you: you MUST fake your user agent pretending to be some very common browser or you'll have a lot of issues. Really a lot. It is amazing the number of webservers out there which are configured (either out of malice or out of incompetence) in a way that will prevent you from accessing them from Java until you fake your user agent. Commented Oct 9, 2010 at 8:25
  • @SyntaxT3rr0r: Interesting. As a professional programmer I must assert that HttpURLConnection is not a mature class if that were the case. Do you have access to any good resources which can be used to fake a user agent or are those illegal due to child robots? Seems like a slippery slope. I wonder if perhaps the class is intended to be used in conjunction with other objects... Commented Jan 3, 2018 at 18:34

1 Answer 1

Reset to default
2

This can be caused by everything. The server may rely on the presence of certain request parameters. The server may rely on a valid session. The server may sniff the user agent. Etc..etc.. Read the server logs for the cause of the error. Or if you don't have access to server logs, read the body of the error response by HttpURLConnection#getErrorStream(), it may contain the error details.

See also:

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

Thanks for the quick answer. Sorry about the lack of information, I tried to put just the essential and might have forgotten other important things. There is no need of valid session, I've tried several different user-agents. I am passing all needed params. Actually you can type exactly the same web address used as "myURL" in the small piece of code I posted in any regular browser and get a valid answer.
The actual answer is in the server logs and/or the error response body. Go read it. If the 500 is coming from a Servlet (you tagged servlets), then it simply means that it has thrown an Exception. In other words, it's possibly a plain vanilla bug in the Servlet code. If you have full control over it, read the logs and fix it accordingly.
As in my above comment: some documentation regarding resources to handle "everything", including user agent, would improve this answer. I'm going to hit the dock of the bay.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.