0

I'm testing my Spring Boot REST API server with Apache JMeter.

Most of 'Connect Time' is in 10ms, but connection timed out occurs sometime.

JMeter's settings

- Thread Number of Threads : 500 Ramp-Up Period : 50 - HTTP Request Connect timeout : 20000ms Response timeout : 40000ms - Constant timer Thread delay : 1000ms

CentOS7 settings

open files (-n) 65535 max user processes (-u) 65535

Java execution settings

-Xmx8g -Xms4g -Xmn2g -XX:PermSize=512m -XX:SurvivorRatio=16

Spring Boot application.properties Tomcat settings

server.tomcat.connection-timeout=20000 server.tomcat.max-connections=100000 server.tomcat.threads.max=10000 server.tomcat.threads.min-spare=2500 server.tomcat.accept-count=10000

JMeter works normally up to 10,000~20,000 requests, but connection timeout occurs after that.

Starting standalone test @ Fri Apr 16 09:30:29 KST 2021 (1618533029412) Waiting for possible Shutdown/StopTestNow/HeapDump/ThreadDump message on port 4445 summary + 1 in 00:00:03 = 0.4/s Avg: 1319 Min: 1319 Max: 1319 Err: 0 (0.00%) Active: 127 Started: 127 Finished: 0 summary + 8771 in 00:00:28 = 318.5/s Avg: 398 Min: 236 Max: 4191 Err: 0 (0.00%) Active: 500 Started: 500 Finished: 0 summary = 8772 in 00:00:30 = 290.3/s Avg: 398 Min: 236 Max: 4191 Err: 0 (0.00%) summary + 9349 in 00:00:30 = 311.6/s Avg: 534 Min: 237 Max: 3582 Err: 0 (0.00%) Active: 500 Started: 500 Finished: 0 summary = 18121 in 00:01:00 = 300.9/s Avg: 468 Min: 236 Max: 4191 Err: 0 (0.00%) summary + 5872 in 00:00:30 = 195.7/s Avg: 1494 Min: 235 Max: 20855 Err: 10 (0.17%) Active: 500 Started: 500 Finished: 0 summary = 23993 in 00:01:30 = 265.9/s Avg: 719 Min: 235 Max: 20855 Err: 10 (0.04%) 
org.apache.http.conn.HttpHostConnectException: Connect to <IP Address>:<Port> [/112.220.184.107] failed: Connection timed out: connect at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:156) at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl$JMeterDefaultHttpClientConnectionOperator.connect(HTTPHC4Impl.java:336) at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:843) at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:574) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:67) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1231) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1220) at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:622) at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:546) at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:486) at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:253) at java.lang.Thread.run(Thread.java:748) Caused by: java.net.ConnectException: Connection timed out: connect at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method) at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:607) at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:75) at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ... 19 more

CPU Request Transaction Thread

Can you tell me why it happens?

Improve this question
1
  • 2
    check for open file descriptors, does JMeter create a new connection all the time (does not seem so), check gc pause Commented Apr 16, 2021 at 0:50

1 Answer 1

Reset to default
2 
summary + 5872 in 00:00:30 = 195.7/s Avg: 1494 Min: 235 Max: 20855 Err: 10 (0.17%) Active: 500 Started: 500 Finished: 0 ^^^^^^^

Your response time increases above 20 seconds hence Tomcat rejects the connection (most probably the setting acts as the protection from slow HTTP attacks

either set it to -1 to disable it or use a profiler tool to see what is causing the problem on your application/tomcat end.

Also JMeter command-line output doesn't tell the full story, I would recommend generating some charts from the .jtl results file like Response Times Over Time, Active Threads Over Time, Transactions per Second, etc. so you could correlate various performance metrics. It's also a good idea to monitor server-side operating system health metrics like CPU, RAM, Network, Swap, Disk usage as well as JVM JMX metrics, it can be done using i.e. JMeter PerfMon Plugin

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I've uploaded my JMeter PerfMon results, but I can't find why it happens..

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.