Spring, Java : Streaming file download for avoiding out of memory errors

Question

I am working on a Spring-MVC application in which users are able to download files. The users can click on an attachment which triggers a download mechanism.

Yesterday, when multiple downloads and two of which had approximately 2 GB files, it caused an out of memory error(log below).

To avoid this problem, one way to solve this problem seemed like streaming the download data in chunks, and only processing those chunks in Service layer, rather than the entire file.

Unfortunately, I don't know how to move ahead with this, any help would be nice. If this option can't fly, any recommendations on how to solve this problem.

Error log :

HTTP Status 500 - Handler processing failed; nested exception is java.lang.OutOfMemoryError: Direct buffer memory type Exception report message Handler processing failed; nested exception is java.lang.OutOfMemoryError: Direct buffer memory description The server encountered an internal error that prevented it from fulfilling this request. exception org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Direct buffer memory org.springframework.web.servlet.DispatcherServlet.triggerAfterCompletionWithError(DispatcherServlet.java:1303) org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:977) org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:893) org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967) org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858) javax.servlet.http.HttpServlet.service(HttpServlet.java:620) org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)

Controller code :

 @RequestMapping(value = "/download/attachment/{attachid}", method = RequestMethod.GET) public void getAttachmentFromDatabase(@PathVariable("attachid") int attachid, , HttpServletResponse response,) { response.setContentType("application/octet-stream"); GroupAttachments groupAttachments = this.groupAttachmentsService.getAttachmenById(attachid); response.setHeader("Content-Disposition", "attachment; filename=\"" + groupAttachments.getFileName() + "\""); response.setContentLength(groupAttachments.getSendAttachment().length); FileCopyUtils.copy(groupAttachments.getSendAttachment(), response.getOutputStream()); response.flushBuffer(); }

Service layer :

@Override public GroupAttachments getAttachmenById(int attachId) { Person person = this.personService.getCurrentlyAuthenticatedUser(); GroupAttachments groupAttachments = this.groupAttachmentsDAO.getAttachmenById(attachId); GroupMembers groupMembers = this.groupMembersService.returnMembersMatchingUsernameAccountId(person.getUsername(), groupAttachments.getGroupId()); if (!(groupMembers == null)) { if (person.getUsername().equals(groupMembers.getMemberUsername())) { try { Path path = Paths.get(msg + groupAttachments.getGroupId() + "/" + groupAttachments.getFileIdentifier()); groupAttachments.setSendAttachment(Files.readAllBytes(path)); return groupAttachments; } catch (IOException ignored) { this.groupAttachmentsDAO.removeAttachment(attachId); return null; } } return null; } else { return null; } }

Thank you. :-)

Update

New Download mechanism :

Controller :

 public ResponseEntity<byte[]> getAttachmentFromDatabase(@PathVariable("attachid") int attachid, @PathVariable("groupaccountid") Long groupAccountId, @PathVariable("api") String api, HttpServletResponse response, @PathVariable("type") boolean type) { Path path = this.groupAttachmentsService.getAttachmentPathById(attachid); GroupAttachments groupAttachments = this.groupAttachmentsService.getAttachmentObjectOnlyById(attachid); response.setContentType("application/octet-stream"); response.setHeader("Content-Disposition", "attachment; filename=\""+groupAttachments.getFileName()+"\""); try { OutputStream outputStream = response.getOutputStream(); Files.copy(path,outputStream); outputStream.flush(); outputStream.close(); response.flushBuffer(); }

Service layer :

@Override public Path getAttachmentPathById(int attachId){ Person person = this.personService.getCurrentlyAuthenticatedUser(); GroupAttachments groupAttachments = this.groupAttachmentsDAO.getAttachmenById(attachId); GroupMembers groupMembers = this.groupMembersService.returnMembersMatchingUsernameAccountId(person.getUsername(), groupAttachments.getGroupId()); if (!(groupMembers == null)) { if (person.getUsername().equals(groupMembers.getMemberUsername())) { try { return Paths.get(msg + groupAttachments.getGroupId() + "/" + groupAttachments.getFileIdentifier()); } catch (Exception ignored) { return null; } } return null; } else { return null; } }

You need to 'chunk' the file as per: Serving large files Spring MVC — Lee
– Lee, Commented May 12, 2016 at 8:45
You are reading everything into memory, don't do that... The Files.readAllBytes loads the whole 2Gb into memory. Simply don't. Delay the loading and instead of first loading it, read it part by part and stream it directly. Use Files.copy(Path, OutputStream) instead. — M. Deinum
– M. Deinum, Commented May 12, 2016 at 8:49
Copy will stream it instead of loading it. Please read the comment... — M. Deinum
– M. Deinum, Commented May 12, 2016 at 9:03
@M.Deinum : Sorry for the mis-read. I have updated the code and also tried with Files.copy(path, outputStream), works nicely. I have posted the code as update in main post, can you please check if any possible optimization. Thank you. — We are Borg
– We are Borg, Commented May 12, 2016 at 9:16

M. Deinum · Accepted Answer · 2016-05-12 09:14:49Z

First stop loading the whole content in your service, as there you are loading the whole lot of the file content into memory.

Create a method which constructs the Path for the GroupAttachments, I would create that on the GroupAttachments it self.

public class GroupAttachments { public Path getPath() { return Paths.get(msg + getGroupId() + "/" + getFileIdentifier()); } }

Then in your controller simply do

@RequestMapping(value = "/download/attachment/{attachid}", method = RequestMethod.GET) public void getAttachmentFromDatabase(@PathVariable("attachid") int attachid, HttpServletResponse response) { response.setContentType("application/octet-stream"); GroupAttachments groupAttachments = this.groupAttachmentsService.getAttachmenById(attachid); Path path = groupAttachmetns.getPath(); // calculates the java.nio.file.Path response.setHeader("Content-Disposition", "attachment; filename=\"" + path.getFileName() + "\""); response.setContentLength(Files.size(path); Files.copy(path, response.getOutputStream()); response.flushBuffer(); }

There is no need to make it more complex imho.

Other than content-length, everything is the same. Thanks a lot. :-)
Some browsers require a content-length, others work fine without it.
Already added, but I had to cast it, response.setContentLength((int) Files.size(path));

Collectives™ on Stack Overflow

Spring, Java : Streaming file download for avoiding out of memory errors

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related