3

I am encountering a deadlock when using an executor based on virtual threads. The issue arises when I am using 50 threads but not at 20. The issue does not shows up when I am not using virtual threads.

I am looking for insights and recommendations on troubleshooting and resolving the deadlock problem unique to virtual threads, especially considering the absence of this issue when virtual threads are not in use.

private static final ExecutorService taskExecutor =0 Executors.newFixedThreadPool(CONCURRENT_WORKERS);

vs

private static final ExecutorService taskExecutor = Executors.newFixedThreadPool(CONCURRENT_WORKERS, Thread.ofVirtual().factory());

The executor service is wrapped in a CompletionService.

private static final CompletionService<Data> completionService = new ExecutorCompletionService<>(taskExecutor)

The main thread seems to wait indefinitely on the first call completionService.take().

Here is the relevant snippet of the code.

 for (int taskIndex = 0; taskIndex < TASK_COUNT; taskIndex++) { completionService.submit(this::doTask); } log.info("Submitted all {} tasks to the executor, waiting.", TASK_COUNT); try { for (int taskIndex = 0; taskIndex < TASK_COUNT; taskIndex++) { Future<Data> future = completionService.take(); try { var data = future.get(); storeData(data); } catch (ExecutionException e) { log.warn("Error executing task", e); } } }

What could be the cause of the issue?

EDIT: this seems to be caused by an HTTP connection pool issue. I am using httpclient-4.5.14.

Internally this client use synchronized which according to my understanding is an issue for Virtual Threads as it pins the platform thread. I don't know if it can be the cause of the deadlock as I can't seen any error, neither weird stuff in the thread dump.

enter image description here

Trials: I tried to use the -Djdk.tracePinnedThreads=full to check for pinned thread but no luck until now.

Improve this question
10
  • 3
    The implications of a synchronized block executed in a virtual thread are not worse than executing the same synchronized block in a platform thread of a limited pool. You may encounter the same starvation issue. If the number of threads was sufficient when using platform threads, it should be sufficient with virtual threads too. It defeats the entire purpose of virtual threads, to maintain a fixed pool of them, though. Commented Dec 1, 2023 at 12:31
  • 4
    I think you're making a wrong assumption; that just because you only observe this when using virtual threads, that somehow it is connected to virtual threads. Deadlocks are usually very timing sensitive; it may merely be that the differences in context switch behavior or scheduling create new interleavings that are uncommon with platform threads. Virtual threads are threads; they only differ in where their stacks live, and who controls their scheduling (the Java runtime vs the OS.) So this is likely just a plain vanilla deadlock. Commented Dec 2, 2023 at 23:42
  • 1
    I am facing the same issue with apache httpclient5. It’s a simple app with an endpoint that invokes another endpoint and sends back a response. When i turn on virtual threads (it’s just a flag in Spring boot 3.2.0), the app hangs completely with 300 concurrent requests. With virtual threads flag turned off, it just works fine. Commented Dec 6, 2023 at 16:17
  • 1
    @Haresh can you app handle 300 concurrent requests when you don’t use virtual threads? Commented Dec 14, 2023 at 10:10
  • 1
    @Haresh do you have the same problem with the built-in http client? Commented Dec 18, 2023 at 15:22

1 Answer 1

Reset to default
2

Pinning is a lingering effect of using virtual threads and can unfortunately lead to serious deadlock scenarios. I've encountered these deadlocks while using various external libraries like Apache HTTP 4.X client, Loading Cache, and Google Cloud Storage Java-client. To address these issues, I had to either:

Upgrade to a newer version of the library that is compatible with virtual threads Switch to an alternative library that supports virtual threads As for the cause of the deadlocks, I believe they stem from a combination of common lock-based logic and synchronized blocks leading to pinning. However, the occurrence of deadlocks was intermittent, suggesting that the sequence of operations for that particular trace also plays a role.

This LinkedIn post provides some relevant context: https://www.linkedin.com/posts/efagerho_java-activity-7130107883915988992-S_zP/

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.