4

I got this error all of a sudden in production while the application was not under any load.

The issue happened when my code tries to send the PUT message using spring rest template

Here is the code how I am initialing the restTemplate

private static final RestTemplate restTemplate = new RestTemplate(new HttpComponentsClientHttpRequestFactory()); { List<HttpMessageConverter<?>> messageConverters = new ArrayList<HttpMessageConverter<?>>(); Jaxb2Marshaller marshaller = new Jaxb2Marshaller(); marshaller.setClassesToBeBound(PaymentSession.class); MarshallingHttpMessageConverter marshallingHttpMessageConverter = new MarshallingHttpMessageConverter(marshaller, marshaller); marshallingHttpMessageConverter.setSupportedMediaTypes(Arrays.asList(MediaType.APPLICATION_XML, MediaType.TEXT_HTML)); messageConverters.add(marshallingHttpMessageConverter); restTemplate.setMessageConverters(messageConverters); }

Call to PUT

try { HttpHeaders headers = new HttpHeaders(); headers.setContentType(MediaType.APPLICATION_XML); HttpEntity<PaymentSession> httpEntity = new HttpEntity<PaymentSession>(session, headers); restTemplate.exchange(baseUrl+"/v1/psps", HttpMethod.PUT, httpEntity, PaymentSession.class); }catch(HttpClientErrorException e){ logger.error("Exception..!!",e) }

Exception stacktrace

Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool at org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:232) at org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:199) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:456) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:88) at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:46) at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:49) at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:509)
Improve this question

3 Answers 3

Reset to default
12

I would suggest to configure HttpComponentsClientHttpRequestFactory instance being passed in the constructor of RestTemplate increasing defaultMaxPerRoute or maxPerRoute for the specific http route for which requests are timing out, increasing the pool size is not enough, as I mentioned in a comment, even if you set PoolingHttpClientConnectionManager.setMaxTotal() to 200, HttpComponentsClientHttpRequestFactory uses a defaultMaxPerRoute of 4, my guess would be in an attempt for a host route (scheme, host, port) not to hijack the connection pool)

... public PoolingHttpClientConnectionManager poolingHttpClientConnectionManager() { PoolingHttpClientConnectionManager result = new PoolingHttpClientConnectionManager(); result.setMaxTotal(this.httpHostConfiguration.getMaxTotal()); // Default max per route is used in case it's not set for a specific route result.setDefaultMaxPerRoute(this.httpHostConfiguration.getDefaultMaxPerRoute()); // and / or if (CollectionUtils.isNotEmpty(this.httpHostConfiguration.getMaxPerRoutes())) { for (HttpHostConfiguration httpHostConfig : this.httpHostConfiguration.getMaxPerRoutes()) { HttpHost host = new HttpHost(httpHostConfig.getHost(), httpHostConfig.getPort(), httpHostConfig.getScheme()); // Max per route for a specific host route result.setMaxPerRoute(new HttpRoute(host), httpHostConfig.getMaxPerRoute()); } } return result; } ... @Configuration @ConfigurationProperties(prefix = "httpConnPool") public class HttpHostsConfiguration { private Integer maxTotal; private Integer defaultMaxPerRoute; private List<HttpHostConfiguration> maxPerRoutes; // Getters, Setters ...

application.yml

httpConnPool: maxTotal: 20 defaultMaxPerRoute: 20 maxPerRoutes: - scheme: http host: localhost port: 8800 maxPerRoute: 20

I recently blog about Troubleshooting Spring's RestTemplate Requests Timeout where requests timing out were troubleshooted using JMeter and shell commands and fixed via configuration settings.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

10

Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool

This error is self describing. You need to increase your connection pool in production - current implementation of HttpComponentsClientHttpRequestFactory default constructor is using HttpClientBuilder with .useSystemProperties().

I believe it will be 5 connections by default. This works for client but is unlikely what you want in server environment. You need to use something like

new RestTemplate(new HttpComponentsClientHttpRequestFactory(HttpClientBuilder.create() .setMaxConnTotal(200) .setMaxConnPerRoute(50) .build()));
Improve this answer

6 Comments

I suspect that this is the issue with connection pool size as I mentioned that the issue occurred under very minimal load and we didn't faced this when we did the load test for 50 concurrent users. But yes we are going to increase the pool size from 5 to 25 (4 servers i.e. 100 connections)
@NullPointerException please set the timeout values also - it may be that in case of failure connection takes several seconds so it goes out of the pool.
@NullPointerException 30 seconds is way too high - 2-3 seconds is what I can imagine - if the servers are in the different parts of the Earth.
One thing to take into account is that it doesn't matter how big is the pool if you don't set .setMaxConnPerRoute(), which I think is by default 4, it will only use 4 connections for the same host, even though you might have another 196 connections idle, I guess to prevent a host from hijacking the pool.
@BorisTreukhov No it is not too high. There is ConnectTimeout (how much time I need for connection establishment) and ReadTimeout (how much should I wait for the request completion). If I read 10MB on the dialup link (just example), then 30secs for ReadTimeout is maybe too low.
|
2

The problem is the HTTP client connections aren't being closed. I had the same problem with a service that only has about 1 request per second. 

"exception":"org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool"

You need to add a finally block and close the connection.

Improve this answer

1 Comment

how to do it if it's apache client's exception, and we're using RestTemplate wrapper?

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.