35

It seems like it the Spring RestTemplate isn't able to stream a response directly to file without buffering it all in memory. What is the proper to achieve this using the newer Spring 5 WebClient?

WebClient client = WebClient.create("https://example.com"); client.get().uri(".../{name}", name).accept(MediaType.APPLICATION_OCTET_STREAM) ....? 

I see people have found a few workarounds/hacks to this issue with RestTemplate, but I am more interested in doing it the proper way with the WebClient.

There are many examples of using RestTemplate to download binary data but almost all of them load the byte[] into memory.

14
  • 1
    Thanks but that doesn’t show how to do it using WebClient. Commented May 19, 2019 at 17:17
  • 1
    I don’t think it answers the question. Please create an answer if you think it does. Commented May 19, 2019 at 17:23
  • 1
    Possible duplicate of Spring WebFlux Webclient receiving an application/octet-stream file as a Mono Commented May 19, 2019 at 18:09
  • 1
    @K.Nicholas - Do you really think this is a duplicate of that question? For one thing that question doesn't mention streaming directly to the file (not keeping the whole response in memory), which is the main point of my question; and also that question is using Kotlin, not Java. Commented May 20, 2019 at 18:03
  • 2
    > Yea, you're right, should have flagged it as off topic. @K.Nicholas I'm not sure why you keep trying to find a way to undermine my question, but feel free to review stackoverflow.com/help/on-topic and the code of conduct. Commented May 20, 2019 at 22:04

4 Answers 4

25

With recent stable Spring WebFlux (5.2.4.RELEASE as of writing):

final WebClient client = WebClient.create("https://example.com"); final Flux<DataBuffer> dataBufferFlux = client.get() .accept(MediaType.TEXT_HTML) .retrieve() .bodyToFlux(DataBuffer.class); // the magic happens here final Path path = FileSystems.getDefault().getPath("target/example.html"); DataBufferUtils .write(dataBufferFlux, path, CREATE_NEW) .block(); // only block here if the rest of your code is synchronous 

For me the non-obvious part was the bodyToFlux(DataBuffer.class), as it is currently mentioned within a generic section about streaming of Spring's documentation, there is no direct reference to it in the WebClient section.

Sign up to request clarification or add additional context in comments.

2 Comments

DataBuffer basically looks just like a ByteBuffer. I don't see any coordination happening between reader & writer, nor any way to set a size limit on the buffer. How do you know that DataBuffer is reactive (or multi-thread coordinated) & size-bounded?
Never mind, apparently the stream generates as many DataBuffers as it needs to, each one containing a chunk of the response data. Not sure how the size is determined, maybe in the Netty config. And each DataBuffer is complete when it's emitted to the Flux so no coordination is needed.
3

I cannot test whether or not the following code effectively does not buffer the contents of webClient payload in memory. Nevertheless, i think you should start from there:

public Mono<Void> testWebClientStreaming() throws IOException { Flux<DataBuffer> stream = webClient .get().accept(MediaType.APPLICATION_OCTET_STREAM) .retrieve() .bodyToFlux(DataBuffer.class); Path filePath = Paths.get("filename"); AsynchronousFileChannel asynchronousFileChannel = AsynchronousFileChannel.open(filePath, WRITE); return DataBufferUtils.write(stream, asynchronousFileChannel) .doOnNext(DataBufferUtils.releaseConsumer()) .doAfterTerminate(() -> { try { asynchronousFileChannel.close(); } catch (IOException ignored) { } }).then(); } 

1 Comment

1

Store the body to a temporary file and consume

static <R> Mono<R> writeBodyToTempFileAndApply( final WebClient.ResponseSpec spec, final Function<? super Path, ? extends R> function) { return using( () -> createTempFile(null, null), t -> write(spec.bodyToFlux(DataBuffer.class), t) .thenReturn(function.apply(t)), t -> { try { deleteIfExists(t); } catch (final IOException ioe) { throw new RuntimeException(ioe); } } ); } 

Pipe the body and consume

static <R> Mono<R> pipeBodyAndApply( final WebClient.ResponseSpec spec, final ExecutorService executor, final Function<? super ReadableByteChannel, ? extends R> function) { return using( Pipe::open, p -> { final Future<Disposable> future = executor.submit( () -> write(spec.bodyToFlux(DataBuffer.class), p.sink()) .log() .doFinally(s -> { try { p.sink().close(); log.debug("p.sink closed"); } catch (final IOException ioe) { throw new RuntimeException(ioe); } }) .subscribe(DataBufferUtils.releaseConsumer()) ); return just(function.apply(p.source())) .log() .doFinally(s -> { try { final Disposable disposable = future.get(); assert disposable.isDisposed(); } catch (InterruptedException | ExecutionException e) { e.printStackTrace(); } }); }, p -> { try { p.source().close(); log.debug("p.source closed"); } catch (final IOException ioe) { throw new RuntimeException(ioe); } } ); } 

2 Comments

Isn't this answer a duplicate of stackoverflow.com/a/56096484/839733?
@AbhijitSarkar Yes.
-1

I'm not sure if you have access to RestTemplate in your current usage of spring, but this one have worked for me.

 RestTemplate restTemplate // = ...; RequestCallback requestCallback = request -> request.getHeaders() .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL)); // Streams the response ResponseExtractor<Void> responseExtractor = response -> { // Here I write the response to a file but do what you like Path path = Paths.get("http://some/path"); Files.copy(response.getBody(), path); return null; }; restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor); 

3 Comments

Thanks for the response. I see the code is the same as from here: stackoverflow.com/a/38664475/449652. However, that doesn't work with Spring 5 or newer anymore -- see this issue: github.com/spring-projects/spring-framework/issues/19448
Original poster said he'd want to do it in the proper way, with Webclient that's non blocking.
RestTemplate is soon going to be deprecated.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.