Download a file with HTTPClient in Java

Question

I'm trying to write a java program that logs into a website, types in a search engine, gets the result, and then downloads the excel file that is generated from the results. So far, I can log in ok. and send a search in and get the results. However, I'm having a lot of problems downloading the excel file.

Looking at the website's source code, I see Ajax and Javascript around the excel file, so I'm assuming it's ajax that helps produce it.

<input id="toexcel" type="image" src="/websmart/v9.4/XLGP/images/Excel-icon.png" alt="To Excel" title="To Excel: Max 20000 Records" onclick="" />

The JavaScript part:

$( document ).ready(function() { $('#toexcel').click(function(e) { e.preventDefault(); setTask('toexcel'); var ajaxForm = $("#filter-form"); $(".spinner").show(); var dataToSend = ajaxForm.serialize(); $("#excelFrame").attr('src','V7BAE01R.pgm' + '?' + dataToSend); setTimeout(function() { $(".spinner").hide(); }, 5000 );

Using TamperData, when I click the Excel File Export, it sends a post request (which I manage to send in the last part of the code) but I'm not sure where to Get it. I do see in tamperdata the Get that says Application/vnd.ms-excel

I'm not sure what to do to add in the code to get the excel file. Below, I tried to use BufferReader, but it doesn't get my file. Some of the code I simplified because of the Name Value pairs.

import java.util.List; import java.util.ArrayList; import org.apache.http.*; import java.io.*; import org.apache.http.client.ClientProtocolException; import org.apache.http.client.methods.*; import org.apache.http.impl.client.*; import org.apache.http.message.*; import org.apache.http.util.EntityUtils; import org.apache.http.client.entity.*; public class httpClientTest { public static void main (String[] args) throws ClientProtocolException, IOException { //Set up HttpClient CloseableHttpClient httpclient = HttpClients.createDefault(); HttpGet httpGet = new HttpGet("http://website"); CloseableHttpResponse response = httpclient.execute(httpGet); //Create Post request to log into the AS400 website HttpPost httpPost = new HttpPost("http://loginwebsite"); List <NameValuePair> nvps = new ArrayList <NameValuePair>(); nvps.add(new BasicNameValuePair("user","username")); nvps.add(new BasicNameValuePair("password","password")); nvps.add(new BasicNameValuePair("button", "Login")); nvps.add(new BasicNameValuePair("task", "extlogin")); httpPost.setEntity(new UrlEncodedFormEntity(nvps)); response = httpclient.execute(httpPost); //Get Post response to ensure we logged in, which succeeds try{ System.out.println(response.getStatusLine()); HttpEntity entity = response.getEntity(); EntityUtils.consume(entity); } finally{ response.close(); } //Sent a Post request to filters out recoreds. httpPost = new HttpPost("http://searchresults"); nvps.clear(); nvps.add(new BasicNameValuePair("ActSts", "Edit")); nvps.add(new BasicNameValuePair("task", "filter")); nvps.add(new BasicNameValuePair("Field", "Plant")); response = httpclient.execute(httpPost); //Displays in printline the html/js of the page. This looks like it DOES display the search results //So it IS sending the Post request and receiving a response. BufferedReader rd = new BufferedReader(new InputStreamReader(response.getEntity().getContent())); String line = ""; while ((line = rd.readLine()) != null) { System.out.println(line); } //try to buffer to read in. String link = "http://website.com/uri?ActSts=Edit&task=filter&Field=Plant"; HttpGet get = new HttpGet(link); response = httpclient.execute(get); InputStream is = response.getEntity().getContent(); String filePath = "C:\\Users\\WindowsUserName\\Downloads\\WODETAIL_List.xls"; FileOutputStream fos = new FileOutputStream(new File(filePath)); int inByte; while((inByte = is.read()) != -1) fos.write(inByte); is.close(); fos.close();

I'm pretty sure I'm Posting the data right, but I'm not sure about how to Get the excel file. Could anybody offer some help?

Edit I was able to download a file, but it wasn't the excel file. It was a webpage, and I think it's a little bit of an improvement. (Before, nothing downloaded, it just hanged there) The problem was, I think I need to send an authorization key or a cookie with this get request to download the file.

Edit 2 I've discovered if I just paste to http://website.com/uri?ActSts=Edit&task=filter&Field=Plant in a new tab while logged in, after waiting a little while, I get a link to the excel file. So originally I thought HTTPClient maintains the same cookies throughout as long as the same httpclient is used but apparently it doesn't(?) I guess I have to figure out a way to get a cookie and send it.

that button is inside a form you should look inside that form and see if there are some hidden fields that get sent to the server , why wouldn't you set a proxy between a browser and the server and monitor every thing coming and going to the server — achabahe
– achabahe, Commented Feb 17, 2016 at 16:55
@achabahe I'm using TamperData and copied and copied and pasted all of the fields (name value pairs) that were shown. When I use the above, instead of downloading a xls file, I download a html file instead. If I open it, it shows me I need to log in, which makes me think I need to find a way to send my login credentials with the GET request to download the file, but after googling all day, and just plain moving code around, I still can't get it to work. — arsarc
– arsarc, Commented Feb 17, 2016 at 21:29

Thomas K. · Accepted Answer · 2020-09-16 13:44:09Z

Oh my god I finally got something that worked. Ok. So apparently HTTPClient can only handle 2 responses before it starts to bug out, according to here: Why does me use HttpClients.createDefault() as HttpClient singleton instance execute third request always hang

So instead, I changed my code to just get a login response, then get the excel file as a response and then quit. I also added some timeout configurations and also changed order from Exporting the file first and then Consuming the entity. I used a separate 2nd response and 2nd entity. That seemed to have helped a bit too? I'm guessing.

import java.util.List; import java.util.ArrayList; import org.apache.http.*; import java.io.*; import org.apache.http.client.ClientProtocolException; import org.apache.http.client.CookieStore; import org.apache.http.client.config.CookieSpecs; import org.apache.http.client.config.RequestConfig; import org.apache.http.client.methods.*; import org.apache.http.client.protocol.HttpClientContext; import org.apache.http.conn.ConnectionPoolTimeoutException; import org.apache.http.cookie.Cookie; import org.apache.http.impl.client.*; import org.apache.http.message.*; import org.apache.http.util.EntityUtils; import org.apache.http.client.entity.*; public class hcFeb { public static void main (String[] args) throws ClientProtocolException, IOException { //Set up Cookie settings and also Timeout settings CookieStore cookieStore = new BasicCookieStore(); HttpClientContext context = HttpClientContext.create(); context.setCookieStore(cookieStore); int CONNECTION_TIMEOUT = 80000; RequestConfig requestConfig = RequestConfig.custom().setCookieSpec(CookieSpecs.DEFAULT) .setConnectionRequestTimeout(CONNECTION_TIMEOUT) .setConnectTimeout(CONNECTION_TIMEOUT) .setSocketTimeout(CONNECTION_TIMEOUT) .build(); //Set up HttpClient CloseableHttpClient httpclient = HttpClients.custom().setDefaultRequestConfig(requestConfig).setDefaultCookieStore(cookieStore).disableContentCompression().build(); HttpGet httpGet = new HttpGet("http://website"); CloseableHttpResponse response = httpclient.execute(httpGet); //Create Post request to log into the website HttpPost httpPost = new HttpPost("http://loginwebsite"); //Login to website List <NameValuePair> nvps = new ArrayList <NameValuePair>(); nvps.add(new BasicNameValuePair("user","username")); nvps.add(new BasicNameValuePair("password","password")); nvps.add(new BasicNameValuePair("button", "Login")); nvps.add(new BasicNameValuePair("task", "extlogin")); httpPost.setEntity(new UrlEncodedFormEntity(nvps)); response = httpclient.execute(httpPost); try{ System.out.println(response.getStatusLine()); HttpEntity entity = response.getEntity(); EntityUtils.consume(entity); } finally{ } //Send request for Excel file and download it. String link = "http://website.com/uri?ActSts=Edit&task=filter&Field=Plant"; HttpGet get = new HttpGet(link); //maybe create new response HttpResponse response2; try{ response2 = httpclient.execute(get,context); System.out.println(response2.getStatusLine()); HttpEntity entity1 = response2.getEntity(); if (entity1 != null) { System.out.println("Entity isn't null"); InputStream is = entity1.getContent(); String filePath = "C:\\Users\\windowsUserName\\Downloads\\WODETAIL_List.xls"; FileOutputStream fos = new FileOutputStream(new File(filePath)); byte[] buffer = new byte[5600]; int inByte; while((inByte = is.read(buffer)) > 0) fos.write(buffer,0,inByte); is.close(); fos.close(); System.out.println("Excel File recieved"); EntityUtils.toString(response2.getEntity()); EntityUtils.consume(entity1); } } catch (ConnectionPoolTimeoutException e){ //response.close(); System.out.println(e.getMessage()); } catch (IOException e){ System.out.println(e.getMessage()); } } }

Collectives™ on Stack Overflow

Download a file with HTTPClient in Java

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related