How to locate thread leak in spring + tomcat application
I have a spring boot (1.5.2) web application that runs in tomcat (8.5) as a sole web app. The total number of threads in the JVM (openjdk 1.8.0_181) increases monotonously almost (though not completely) at a constant rate, going from a few hundred in the beginning, to about 3000 in a week. By that time, the majority of the threads stack traces look like:
WAITING Thread[pool-917-thread-1,5,main]
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
The application works otherwise fine, these threads do not seem to take up too much memory (at least compared to the tens of gigabytes the application normally consumes), but their existence points to some hidden leak inside the application. As of writing this, I can not currently find a thread named pool-.* in a different state, so I don't know what they normally do before going zombie. The application is never redeployed without a tomcat restart.
My question would be whether anyone encountered anything similar and how they solved it, and if not, how could I diagnose why these threads are being created and not removed afterwards.
java spring tomcat threadpool
asked yesterday
P.PéterP.Péter
892929
add a comment |
I have a spring boot (1.5.2) web application that runs in tomcat (8.5) as a sole web app. The total number of threads in the JVM (openjdk 1.8.0_181) increases monotonously almost (though not completely) at a constant rate, going from a few hundred in the beginning, to about 3000 in a week. By that time, the majority of the threads stack traces look like:
WAITING Thread[pool-917-thread-1,5,main]
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
The application works otherwise fine, these threads do not seem to take up too much memory (at least compared to the tens of gigabytes the application normally consumes), but their existence points to some hidden leak inside the application. As of writing this, I can not currently find a thread named pool-.* in a different state, so I don't know what they normally do before going zombie. The application is never redeployed without a tomcat restart.
My question would be whether anyone encountered anything similar and how they solved it, and if not, how could I diagnose why these threads are being created and not removed afterwards.
java spring tomcat threadpool
asked yesterday
P.PéterP.Péter
892929
So you mean to say all threads are waiting and none have closed. Had it been an issue with tomcat, you would have easily found that out by now, anyways do mentioned the tomcat version you are using. Are you anywhere in your application setting up Executors, if yes. I would recommend checking that code or using the constructor that takes ThreadFactory as input. Override the newThread method in the factory to use custom names. If its your application then with the thread name you can easily figure out the faulty code and work on that particular piece.
– Himanshu Bhardwaj
yesterday
OK so it's a worker thread pool. What are the thread names?
– Andy Brown
yesterday
@AndyBrown thread names:pool-[0-9]*-thread-1,5,main
– P.Péter
yesterday
add a comment |
I have a spring boot (1.5.2) web application that runs in tomcat (8.5) as a sole web app. The total number of threads in the JVM (openjdk 1.8.0_181) increases monotonously almost (though not completely) at a constant rate, going from a few hundred in the beginning, to about 3000 in a week. By that time, the majority of the threads stack traces look like:
WAITING Thread[pool-917-thread-1,5,main]
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
The application works otherwise fine, these threads do not seem to take up too much memory (at least compared to the tens of gigabytes the application normally consumes), but their existence points to some hidden leak inside the application. As of writing this, I can not currently find a thread named pool-.* in a different state, so I don't know what they normally do before going zombie. The application is never redeployed without a tomcat restart.
My question would be whether anyone encountered anything similar and how they solved it, and if not, how could I diagnose why these threads are being created and not removed afterwards.
java spring tomcat threadpool
asked yesterday
P.PéterP.Péter
892929
I have a spring boot (1.5.2) web application that runs in tomcat (8.5) as a sole web app. The total number of threads in the JVM (openjdk 1.8.0_181) increases monotonously almost (though not completely) at a constant rate, going from a few hundred in the beginning, to about 3000 in a week. By that time, the majority of the threads stack traces look like:
WAITING Thread[pool-917-thread-1,5,main]
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
The application works otherwise fine, these threads do not seem to take up too much memory (at least compared to the tens of gigabytes the application normally consumes), but their existence points to some hidden leak inside the application. As of writing this, I can not currently find a thread named pool-.* in a different state, so I don't know what they normally do before going zombie. The application is never redeployed without a tomcat restart.
My question would be whether anyone encountered anything similar and how they solved it, and if not, how could I diagnose why these threads are being created and not removed afterwards.
java spring tomcat threadpool
java spring tomcat threadpool
asked yesterday
P.PéterP.Péter
892929
asked yesterday
P.PéterP.Péter
892929
edited yesterday
P.Péter
asked yesterday
P.PéterP.Péter
892929
asked yesterday
P.PéterP.Péter
892929
asked yesterday
P.PéterP.Péter
892929
892929
So you mean to say all threads are waiting and none have closed. Had it been an issue with tomcat, you would have easily found that out by now, anyways do mentioned the tomcat version you are using. Are you anywhere in your application setting up Executors, if yes. I would recommend checking that code or using the constructor that takes ThreadFactory as input. Override the newThread method in the factory to use custom names. If its your application then with the thread name you can easily figure out the faulty code and work on that particular piece.
– Himanshu Bhardwaj
yesterday
OK so it's a worker thread pool. What are the thread names?
– Andy Brown
yesterday
@AndyBrown thread names:pool-[0-9]*-thread-1,5,main
– P.Péter
yesterday
add a comment |
So you mean to say all threads are waiting and none have closed. Had it been an issue with tomcat, you would have easily found that out by now, anyways do mentioned the tomcat version you are using. Are you anywhere in your application setting up Executors, if yes. I would recommend checking that code or using the constructor that takes ThreadFactory as input. Override the newThread method in the factory to use custom names. If its your application then with the thread name you can easily figure out the faulty code and work on that particular piece.
– Himanshu Bhardwaj
yesterday
OK so it's a worker thread pool. What are the thread names?
– Andy Brown
yesterday
@AndyBrown thread names:pool-[0-9]*-thread-1,5,main
– P.Péter
yesterday
So you mean to say all threads are waiting and none have closed. Had it been an issue with tomcat, you would have easily found that out by now, anyways do mentioned the tomcat version you are using. Are you anywhere in your application setting up Executors, if yes. I would recommend checking that code or using the constructor that takes ThreadFactory as input. Override the newThread method in the factory to use custom names. If its your application then with the thread name you can easily figure out the faulty code and work on that particular piece.
– Himanshu Bhardwaj
yesterday
So you mean to say all threads are waiting and none have closed. Had it been an issue with tomcat, you would have easily found that out by now, anyways do mentioned the tomcat version you are using. Are you anywhere in your application setting up Executors, if yes. I would recommend checking that code or using the constructor that takes ThreadFactory as input. Override the newThread method in the factory to use custom names. If its your application then with the thread name you can easily figure out the faulty code and work on that particular piece.
– Himanshu Bhardwaj
yesterday
OK so it's a worker thread pool. What are the thread names?
– Andy Brown
yesterday
OK so it's a worker thread pool. What are the thread names?
– Andy Brown
yesterday
@AndyBrown thread names:
pool-[0-9]*-thread-1,5,main– P.Péter
yesterday
@AndyBrown thread names:
pool-[0-9]*-thread-1,5,main– P.Péter
yesterday
add a comment |
1 Answer
1
active
oldest
votes
The most likely causes for threads being spawned in this fashion:
- Tomcat was misconfigured e.g. executor
maxThreadsset to unreasonably high value from default 200. - Application code creates an ubound thread pool e.g.
Executors.newCachedThreadPool()which experiences a thread spike. - During application re-deployment the thread pool created by undeployed application is not stopped correctly. See this answer.
- Application code creates threads with
new Thread().
Point 4 is unlikely since your stack trace is showing java.util.concurrent.ThreadPoolExecutor. Find the pool which creates threads with prefix pool- (grep application code and Tomcat configuration for pool-). Then cap the pool.
answered yesterday
Karol DowbeckiKarol Dowbecki
19.6k92851
1
Point 1 is not likely. Tomcat uses it's ownTaskQueueclass extendingLinkedBlockingQueueand overridingtake(). TaskQueue.take() would appear in the stack trace.
– Andy Brown
yesterday
Point 3 is not likely, we always restart tomcat before redeploying. I will edit the question to add additional info.
– P.Péter
yesterday
Point 1 is also not likely, as those should start with catalina-exec- as server.xml states:<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"Oh sorry, it is commented out, so could be that.
– P.Péter
yesterday
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54251799%2fhow-to-locate-thread-leak-in-spring-tomcat-application%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
The most likely causes for threads being spawned in this fashion:
- Tomcat was misconfigured e.g. executor
maxThreadsset to unreasonably high value from default 200. - Application code creates an ubound thread pool e.g.
Executors.newCachedThreadPool()which experiences a thread spike. - During application re-deployment the thread pool created by undeployed application is not stopped correctly. See this answer.
- Application code creates threads with
new Thread().
Point 4 is unlikely since your stack trace is showing java.util.concurrent.ThreadPoolExecutor. Find the pool which creates threads with prefix pool- (grep application code and Tomcat configuration for pool-). Then cap the pool.
answered yesterday
Karol DowbeckiKarol Dowbecki
19.6k92851
1
Point 1 is not likely. Tomcat uses it's ownTaskQueueclass extendingLinkedBlockingQueueand overridingtake(). TaskQueue.take() would appear in the stack trace.
– Andy Brown
yesterday
Point 3 is not likely, we always restart tomcat before redeploying. I will edit the question to add additional info.
– P.Péter
yesterday
Point 1 is also not likely, as those should start with catalina-exec- as server.xml states:<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"Oh sorry, it is commented out, so could be that.
– P.Péter
yesterday
add a comment |
The most likely causes for threads being spawned in this fashion:
- Tomcat was misconfigured e.g. executor
maxThreadsset to unreasonably high value from default 200. - Application code creates an ubound thread pool e.g.
Executors.newCachedThreadPool()which experiences a thread spike. - During application re-deployment the thread pool created by undeployed application is not stopped correctly. See this answer.
- Application code creates threads with
new Thread().
Point 4 is unlikely since your stack trace is showing java.util.concurrent.ThreadPoolExecutor. Find the pool which creates threads with prefix pool- (grep application code and Tomcat configuration for pool-). Then cap the pool.
answered yesterday
Karol DowbeckiKarol Dowbecki
19.6k92851
1
Point 1 is not likely. Tomcat uses it's ownTaskQueueclass extendingLinkedBlockingQueueand overridingtake(). TaskQueue.take() would appear in the stack trace.
– Andy Brown
yesterday
Point 3 is not likely, we always restart tomcat before redeploying. I will edit the question to add additional info.
– P.Péter
yesterday
Point 1 is also not likely, as those should start with catalina-exec- as server.xml states:<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"Oh sorry, it is commented out, so could be that.
– P.Péter
yesterday
add a comment |
The most likely causes for threads being spawned in this fashion:
- Tomcat was misconfigured e.g. executor
maxThreadsset to unreasonably high value from default 200. - Application code creates an ubound thread pool e.g.
Executors.newCachedThreadPool()which experiences a thread spike. - During application re-deployment the thread pool created by undeployed application is not stopped correctly. See this answer.
- Application code creates threads with
new Thread().
Point 4 is unlikely since your stack trace is showing java.util.concurrent.ThreadPoolExecutor. Find the pool which creates threads with prefix pool- (grep application code and Tomcat configuration for pool-). Then cap the pool.
answered yesterday
Karol DowbeckiKarol Dowbecki
19.6k92851
The most likely causes for threads being spawned in this fashion:
- Tomcat was misconfigured e.g. executor
maxThreadsset to unreasonably high value from default 200. - Application code creates an ubound thread pool e.g.
Executors.newCachedThreadPool()which experiences a thread spike. - During application re-deployment the thread pool created by undeployed application is not stopped correctly. See this answer.
- Application code creates threads with
new Thread().
Point 4 is unlikely since your stack trace is showing java.util.concurrent.ThreadPoolExecutor. Find the pool which creates threads with prefix pool- (grep application code and Tomcat configuration for pool-). Then cap the pool.
answered yesterday
Karol DowbeckiKarol Dowbecki
19.6k92851
answered yesterday
Karol DowbeckiKarol Dowbecki
19.6k92851
answered yesterday
Karol DowbeckiKarol Dowbecki
19.6k92851
answered yesterday
Karol DowbeckiKarol Dowbecki
19.6k92851
19.6k92851
1
Point 1 is not likely. Tomcat uses it's ownTaskQueueclass extendingLinkedBlockingQueueand overridingtake(). TaskQueue.take() would appear in the stack trace.
– Andy Brown
yesterday
Point 3 is not likely, we always restart tomcat before redeploying. I will edit the question to add additional info.
– P.Péter
yesterday
Point 1 is also not likely, as those should start with catalina-exec- as server.xml states:<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"Oh sorry, it is commented out, so could be that.
– P.Péter
yesterday
add a comment |
1
Point 1 is not likely. Tomcat uses it's ownTaskQueueclass extendingLinkedBlockingQueueand overridingtake(). TaskQueue.take() would appear in the stack trace.
– Andy Brown
yesterday
Point 3 is not likely, we always restart tomcat before redeploying. I will edit the question to add additional info.
– P.Péter
yesterday
Point 1 is also not likely, as those should start with catalina-exec- as server.xml states:<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"Oh sorry, it is commented out, so could be that.
– P.Péter
yesterday
1
1
Point 1 is not likely. Tomcat uses it's own
TaskQueue class extending LinkedBlockingQueue and overriding take(). TaskQueue.take() would appear in the stack trace.– Andy Brown
yesterday
Point 1 is not likely. Tomcat uses it's own
TaskQueue class extending LinkedBlockingQueue and overriding take(). TaskQueue.take() would appear in the stack trace.– Andy Brown
yesterday
Point 3 is not likely, we always restart tomcat before redeploying. I will edit the question to add additional info.
– P.Péter
yesterday
Point 3 is not likely, we always restart tomcat before redeploying. I will edit the question to add additional info.
– P.Péter
yesterday
Point 1 is also not likely, as those should start with catalina-exec- as server.xml states:
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-" Oh sorry, it is commented out, so could be that.– P.Péter
yesterday
Point 1 is also not likely, as those should start with catalina-exec- as server.xml states:
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-" Oh sorry, it is commented out, so could be that.– P.Péter
yesterday
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f54251799%2fhow-to-locate-thread-leak-in-spring-tomcat-application%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
So you mean to say all threads are waiting and none have closed. Had it been an issue with tomcat, you would have easily found that out by now, anyways do mentioned the tomcat version you are using. Are you anywhere in your application setting up Executors, if yes. I would recommend checking that code or using the constructor that takes ThreadFactory as input. Override the newThread method in the factory to use custom names. If its your application then with the thread name you can easily figure out the faulty code and work on that particular piece.
– Himanshu Bhardwaj
yesterday
OK so it's a worker thread pool. What are the thread names?
– Andy Brown
yesterday
@AndyBrown thread names:
pool-[0-9]*-thread-1,5,main– P.Péter
yesterday