I have a Spring rest service which runs periodically after every 10 min. Service knows the path to a file stored on remote server. Size of the file is around 2GB. Service fetches the file and then reads contents of the file and then manipulates the database based on file contents. What is the best way for spring rest service to fetch large file and operate upon its contents.
1 Answer
Use Apache File utils,
http://commons.apache.org/proper/commons-io/apidocs/org/apache/commons/io/FileUtils.html#copyURLToFile(java.net.URL, java.io.File, int, int)
An example code to download 1 GB file from online is as below,
package com.jai.download; import java.io.File; import java.net.URL; import org.apache.commons.io.FileUtils; public class FileDownload { public static void main(String[] args) { File file = new File("c:/jj/1gb.zip"); URL url; try { url = new URL("http://download.thinkbroadband.com/1GB.zip"); long start = System.currentTimeMillis(); System.out.println("Downloading...."); FileUtils.copyURLToFile(url, file); long end = System.currentTimeMillis(); System.out.println("Completed....in ms : " + (end - start)); } catch (Exception e) { e.printStackTrace(); } } } With my internet speed of 100 Mbps, it downloaded this 1GB file in 94 secs. Once you have your csv file like this downloaded you could read its contents to do you business logic check and make appropriate action.
1 Comment
Shreerang
Its a good option. Only drawback I see is that it does not support Auth (or any custom) Headers that might be required by the URL.
gzipcompression is an idea.