Upgrade to JDK 24, Netty 4.2 and Virtual Thread Experimental support

[franz1981]

This is both upgrading to Netty 4.2, unifying few existing code paths and introducing a novel "feature" for Netty
i.e. event loop as carriers for https://openjdk.org/jeps/444

The feature here is used in an highly opinionated way, ensuring that pipelining is still possible and effective.

[franz1981]

FYI @yawkat

[franz1981]

This is blocked by netty/netty@620794f

[franz1981]

Ok it seems netty/netty@620794f is not as harmfull as expected and we have some decent flamegraphs for this both with/without Loom - although we need to put some extra care while configure it because:

• virtual thread containers tracking uses a concurrent hash map under the hood (see https://github.com/openjdk/jdk/blob/1fc1cc5da9a38cf936636a72f9b8a4c246ceaab4/src/java.base/share/classes/jdk/internal/vm/ThreadContainers.java#L248)
• virtual thread JVMTI state management (while attaching a profiler/agent) can still harms, but only while profiling (see [Loom] +DoJVMTIVirtualThreadTransitions scalability issue async-profiler/async-profiler#779)

The current Loom test is disabling some of these features to avoid been bitten - but sadly the many atomic operations performed while setting the state and carrier in VirtualThread - with pipelining - can still be a major cause of regression, compared to NOT using virtual threads - see https://github.com/openjdk/jdk/blob/dc249be2474d8711ca6e92ec139b4b6f0aa92e6e/src/java.base/share/classes/java/lang/VirtualThread.java#L1383-L1386

flamegraphs attached - for plaintext:

netty_plaintext.zip

[franz1981]

To clarify the point about Virtual Thread state management, see

and the relative print asm for x86 obtained adding -XX:CompileCommand=print,java.lang.VirtualThread::runContinuation to the server jvm args:

============================= C2-compiled nmethod ============================== ----------------------------------- Assembly ----------------------------------- Compiled method (c2) 2315 2090 ! 4 java.lang.VirtualThread::runContinuation (160 bytes)  total in heap [0x00007feb1c160508,0x00007feb1c160f00] = 2552  relocation [0x00007feb1c1605e0,0x00007feb1c160680] = 160  main code [0x00007feb1c160680,0x00007feb1c160da8] = 1832  stub code [0x00007feb1c160da8,0x00007feb1c160dd0] = 40  oops [0x00007feb1c160dd0,0x00007feb1c160de0] = 16  metadata [0x00007feb1c160de0,0x00007feb1c160f00] = 288  immutable data [0x00007fea884cf500,0x00007fea884d00d8] = 3032  dependencies [0x00007fea884cf500,0x00007fea884cf558] = 88  nul chk table [0x00007fea884cf558,0x00007fea884cf588] = 48  handler table [0x00007fea884cf588,0x00007fea884cf5a0] = 24  scopes pcs [0x00007fea884cf5a0,0x00007fea884cfcd0] = 1840  scopes data [0x00007fea884cfcd0,0x00007fea884d00d8] = 1032 [Disassembly] -------------------------------------------------------------------------------- [Constant Pool (empty)] -------------------------------------------------------------------------------- [Instructions begin]  0x00007feb1c160680: xchg %ax,%ax [Entry Point]  # {method} {0x00007feabb187028} 'runContinuation' '()V' in 'java/lang/VirtualThread'  # [sp+0x50] (sp of caller)  0x00007feb1c160682: mov 0x8(%rsi),%r10d  0x00007feb1c160686: cmp 0x8(%rax),%r10d  0x00007feb1c16068a: jne 0x00007feb1bb85f60 ; {runtime_call Shared Runtime ic_miss_blob} [Verified Entry Point]  0x00007feb1c160690: mov %eax,-0x14000(%rsp)  0x00007feb1c160697: push %rbp  0x00007feb1c160698: sub $0x40,%rsp  0x00007feb1c16069c: cmpl $0x1,0x20(%r15)  0x00007feb1c1606a4: jne 0x00007feb1c160d9b ;*synchronization entry  ; - java.lang.VirtualThread::runContinuation@-1 (line 275)  0x00007feb1c1606aa: mov %rsi,(%rsp)  0x00007feb1c1606ae: mov 0x390(%r15),%r10  0x00007feb1c1606b5: mov (%r10),%r10 ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c1606b8: mov 0x8(%r10),%r11d  0x00007feb1c1606bc: cmp $0x1018248,%r11d ; {metadata('io/netty/util/concurrent/FastThreadLocalThread')}  0x00007feb1c1606c3: jne 0x00007feb1c160cc4 ;*instanceof {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::isVirtual@1 (line 1373)  ; - java.lang.VirtualThread::runContinuation@3 (line 275)  0x00007feb1c1606c9: mov 0x80(%rsi),%r11d ;*getfield state {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::state@1 (line 1350)  ; - java.lang.VirtualThread::runContinuation@18 (line 280)  0x00007feb1c1606d0: cmp $0x1,%r11d  0x00007feb1c1606d4: jne 0x00007feb1c160be8  0x00007feb1c1606da: mov $0x2,%r10d  0x00007feb1c1606e0: mov $0x1,%eax  0x00007feb1c1606e5: lock cmpxchg %r10d,0x80(%rsi)  0x00007feb1c1606ee: sete %bpl  0x00007feb1c1606f2: movzbl %bpl,%ebp ;*invokevirtual compareAndSetInt {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::compareAndSetState@9 (line 1358)  ; - java.lang.VirtualThread::runContinuation@48 (line 284)  0x00007feb1c1606f6: test %ebp,%ebp  0x00007feb1c1606f8: je 0x00007feb1c160c04 ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c1606fe: mov 0x388(%r15),%r10  0x00007feb1c160705: mov (%r10),%r8  0x00007feb1c160708: mov %r8,%r11  0x00007feb1c16070b: shr $0x3,%r11  0x00007feb1c16070f: mov %r11d,0xa0(%rsi)  0x00007feb1c160716: mov %rsi,%r10  0x00007feb1c160719: shr $0x9,%r10  0x00007feb1c16071d: movabs $0x7feb10600000,%r11  0x00007feb1c160727: mov %r12b,(%r11,%r10,1)  0x00007feb1c16072b: lock addl $0x0,-0x40(%rsp) ;*putfield carrierThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::setCarrierThread@2 (line 1381)  ; - java.lang.VirtualThread::mount@11 (line 494)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  0x00007feb1c160731: movzbl 0x3a(%rsi),%ebp ;*getfield interrupted {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@15 (line 497)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  0x00007feb1c160735: test %ebp,%ebp  0x00007feb1c160737: jne 0x00007feb1c160c18 ;*ifeq {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@18 (line 497)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  0x00007feb1c16073d: mov 0x8(%r8),%r10d  0x00007feb1c160741: cmp $0x1018248,%r10d ; {metadata('io/netty/util/concurrent/FastThreadLocalThread')}  0x00007feb1c160748: jne 0x00007feb1c160b38 ;*invokevirtual isInterrupted {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@29 (line 499)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  0x00007feb1c16074e: movzbl 0x3a(%r8),%ebp ;*getfield interrupted {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::isInterrupted@1 (line 1610)  ; - java.lang.VirtualThread::mount@29 (line 499)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  0x00007feb1c160753: test %ebp,%ebp  0x00007feb1c160755: jne 0x00007feb1c160c30 ;*ifeq {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@32 (line 499)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  0x00007feb1c16075b: mov 0x18(%rsi),%r10  0x00007feb1c16075f: mov 0x390(%r15),%r11  0x00007feb1c160766: mov %rsi,(%r11)  0x00007feb1c160769: mov %r10,0x448(%r15)  0x00007feb1c160770: mov 0x388(%r15),%r11  0x00007feb1c160777: cmp (%r11),%rsi  0x00007feb1c16077a: je 0x00007feb1c160afb ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c160780: mov %r10,0x2b0(%r15)  0x00007feb1c160787: movzwl 0x38(%rsi),%r10d  0x00007feb1c16078c: mov %r10d,%r8d  0x00007feb1c16078f: and $0x8000,%r8d  0x00007feb1c160796: jne 0x00007feb1c1607a7 ;*invokevirtual setCurrentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@65 (line 509)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  0x00007feb1c160798: and $0x7fff,%r10d ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c16079f: mov %r10w,0x310(%r15) ;*invokevirtual setCurrentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@65 (line 509)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  0x00007feb1c1607a7: xor %r11d,%r11d  0x00007feb1c1607aa: test %r8d,%r8d  0x00007feb1c1607ad: setne %r11b  0x00007feb1c1607b1: mov %r11b,0x312(%r15) ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c1607b8: movb $0x1,0x314(%r15) ;*invokevirtual setCurrentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@65 (line 509)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  0x00007feb1c1607c0: mov 0x90(%rsi),%r10d ;*getfield cont {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@97 (line 303)  0x00007feb1c1607c7: test %r10d,%r10d  0x00007feb1c1607ca: je 0x00007feb1c160b07 ;*invokevirtual run {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@100 (line 303)  0x00007feb1c1607d0: lea (%r12,%r10,8),%rsi ;*getfield cont {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@97 (line 303)  0x00007feb1c1607d4: data16 xchg %ax,%ax  0x00007feb1c1607d7: call 0x00007feb14824f20 ; ImmutableOopMap {[0]=Oop }  ;*invokevirtual run {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@100 (line 303)  ; {optimized virtual_call}  0x00007feb1c1607dc: nopl 0x2d4(%rax,%rax,1) ;*invokevirtual run {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@100 (line 303)  ; {post_call_nop}  0x00007feb1c1607e4: mov (%rsp),%r10  0x00007feb1c1607e8: mov 0xa0(%r10),%r10d  0x00007feb1c1607ef: mov %r10d,0x18(%rsp) ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c1607f4: mov 0x390(%r15),%r10  0x00007feb1c1607fb: mov 0x18(%rsp),%r8d  0x00007feb1c160800: mov 0x18(%r12,%r8,8),%r11 ;*invokevirtual setCurrentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@31 (line 523)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c160805: shl $0x3,%r8 ;*getfield carrierThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@25 (line 522)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c160809: mov %r8,(%r10)  0x00007feb1c16080c: mov %r11,0x448(%r15)  0x00007feb1c160813: mov 0x388(%r15),%r10  0x00007feb1c16081a: cmp (%r10),%r8  0x00007feb1c16081d: je 0x00007feb1c160aef ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c160823: mov %r11,0x2b0(%r15)  0x00007feb1c16082a: mov 0x18(%rsp),%r11d  0x00007feb1c16082f: movzwl 0x38(%r12,%r11,8),%r11d  0x00007feb1c160835: mov %r11d,%r10d  0x00007feb1c160838: and $0x8000,%r10d  0x00007feb1c16083f: jne 0x00007feb1c160850 ;*invokevirtual setCurrentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@31 (line 523)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c160841: and $0x7fff,%r11d ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c160848: mov %r11w,0x310(%r15) ;*invokevirtual setCurrentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@31 (line 523)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c160850: xor %r11d,%r11d  0x00007feb1c160853: test %r10d,%r10d  0x00007feb1c160856: setne %r11b  0x00007feb1c16085a: mov %r11b,0x312(%r15) ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c160861: movb $0x1,0x314(%r15) ;*invokevirtual setCurrentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@31 (line 523)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c160869: mov (%rsp),%r10  0x00007feb1c16086d: mov 0x54(%r10),%r10d ;*getfield interruptLock {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@35 (line 526)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c160871: test %r10d,%r10d  0x00007feb1c160874: je 0x00007feb1c160b20 ;*monitorenter {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@40 (line 526)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c16087a: lea (%r12,%r10,8),%rbp ;*getfield interruptLock {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@35 (line 526)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c16087e: lea 0x30(%rsp),%rbx  0x00007feb1c160883: mov 0x0(%rbp),%r10  0x00007feb1c160887: mov 0x6f8(%r15),%ebx  0x00007feb1c16088e: test $0x2,%r10b  0x00007feb1c160892: jne 0x00007feb1c1608d0  0x00007feb1c160898: cmp $0x747,%ebx  0x00007feb1c16089e: jg 0x00007feb1c1608f0  0x00007feb1c1608a4: cmp -0x8(%r15,%rbx,1),%rbp  0x00007feb1c1608a9: je 0x00007feb1c1608c2  0x00007feb1c1608ab: mov %r10,%rax  0x00007feb1c1608ae: or $0x1,%rax  0x00007feb1c1608b2: and $0xfffffffffffffffe,%r10  0x00007feb1c1608b6: lock cmpxchg %r10,0x0(%rbp)  0x00007feb1c1608bc: jne 0x00007feb1c1608f0  0x00007feb1c1608c2: mov %rbp,(%r15,%rbx,1)  0x00007feb1c1608c6: addl $0x8,0x6f8(%r15)  0x00007feb1c1608ce: jmp 0x00007feb1c1608ee  0x00007feb1c1608d0: xor %rax,%rax  0x00007feb1c1608d3: mov 0x448(%r15),%rbx  0x00007feb1c1608da: lock cmpxchg %rbx,0x3e(%r10) ; {no_reloc}  0x00007feb1c1608e0: je 0x00007feb1c1608ee  0x00007feb1c1608e2: cmp %rax,%rbx  0x00007feb1c1608e5: jne 0x00007feb1c1608f0  0x00007feb1c1608e7: incq 0x86(%r10)  0x00007feb1c1608ee: xor %eax,%eax  0x00007feb1c1608f0: jne 0x00007feb1c160b54  0x00007feb1c1608f6: mov (%rsp),%r10  0x00007feb1c1608fa: mov %r12d,0xa0(%r10)  0x00007feb1c160901: lock addl $0x0,-0x40(%rsp) ;*monitorexit {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@47 (line 528)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c160907: lea 0x30(%rsp),%rax ;*monitorenter {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@40 (line 526)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c16090c: mov 0x6f8(%r15),%eax  0x00007feb1c160913: mov 0x0(%rbp),%r10  0x00007feb1c160917: cmp -0x8(%r15,%rax,1),%rbp  0x00007feb1c16091c: jne 0x00007feb1c160951  0x00007feb1c160922: subl $0x8,0x6f8(%r15)  0x00007feb1c16092a: cmp -0x10(%r15,%rax,1),%rbp  0x00007feb1c16092f: je 0x00007feb1c16099b  0x00007feb1c160935: mov %r10,%rax  0x00007feb1c160938: and $0xfffffffffffffffc,%rax  0x00007feb1c16093c: or $0x1,%r10  0x00007feb1c160940: lock cmpxchg %r10,0x0(%rbp)  0x00007feb1c160946: jne 0x00007feb1c160d8e  0x00007feb1c16094c: jmp 0x00007feb1c16099b  0x00007feb1c160951: cmpq $0x0,0x86(%r10)  0x00007feb1c160959: jne 0x00007feb1c160994  0x00007feb1c16095b: movq $0x0,0x3e(%r10)  0x00007feb1c160963: lock addl $0x0,-0x40(%rsp)  0x00007feb1c160969: mov 0x8e(%r10),%rax  0x00007feb1c160970: or 0x96(%r10),%rax  0x00007feb1c160977: je 0x00007feb1c16099b  0x00007feb1c160979: cmpq $0x0,0x9e(%r10)  0x00007feb1c160981: jne 0x00007feb1c16099b  0x00007feb1c160983: and $0xfffffffffffffffd,%r10  0x00007feb1c160987: mov %r10,0x568(%r15)  0x00007feb1c16098e: or $0x1,%r10d  0x00007feb1c160992: jmp 0x00007feb1c16099e  0x00007feb1c160994: decq 0x86(%r10)  0x00007feb1c16099b: xor %r10d,%r10d  0x00007feb1c16099e: jne 0x00007feb1c160b75 ;*monitorexit {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@47 (line 528)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c1609a4: mov 0x18(%rsp),%r11d  0x00007feb1c1609a9: movzbl 0x3a(%r12,%r11,8),%ebp ; implicit exception: dispatches to 0x00007feb1c160cfc  ;*getfield interrupted {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::clearInterrupt@1 (line 1623)  ; - java.lang.VirtualThread::unmount@57 (line 529)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c1609af: test %ebp,%ebp  0x00007feb1c1609b1: jne 0x00007feb1c160c48 ;*ifeq {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::clearInterrupt@4 (line 1623)  ; - java.lang.VirtualThread::unmount@57 (line 529)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c1609b7: mov (%rsp),%r10  0x00007feb1c1609bb: mov 0x90(%r10),%r11d ;*getfield cont {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@108 (line 306)  0x00007feb1c1609c2: movzbl 0xc(%r12,%r11,8),%ebp ; implicit exception: dispatches to 0x00007feb1c160d14  ;*getfield done {reexecute=0 rethrow=0 return_oop=0}  ; - jdk.internal.vm.Continuation::isDone@1 (line 420)  ; - java.lang.VirtualThread::runContinuation@111 (line 306)  0x00007feb1c1609c8: test %ebp,%ebp  0x00007feb1c1609ca: je 0x00007feb1c160c5c  0x00007feb1c1609d0: movl $0x63,0x80(%r10) ; {no_reloc}  0x00007feb1c1609db: lock addl $0x0,-0x40(%rsp) ;*putfield state {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::setState@2 (line 1354)  ; - java.lang.VirtualThread::afterDone@24 (line 664)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c1609e1: mov 0xa4(%r10),%ebp ;*getfield termination {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::afterDone@28 (line 667)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c1609e8: test %ebp,%ebp  0x00007feb1c1609ea: jne 0x00007feb1c160c70 ;*ifnull {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::afterDone@33 (line 668)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c1609f0: mov 0x68(%r10),%ebp ;*getfield container {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::threadContainer@1 (line 2572)  ; - java.lang.VirtualThread::afterDone@68 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c1609f4: mov 0x8(%r12,%rbp,8),%r11d ; implicit exception: dispatches to 0x00007feb1c160d2c  0x00007feb1c1609f9: cmp $0x1064460,%r11d ; {metadata('jdk/internal/vm/ThreadContainers$RootContainer$CountingRootContainer')}  0x00007feb1c160a00: jne 0x00007feb1c160b9a ;*invokevirtual onExit {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a06: movabs $0x6b5bd29a0,%r10 ; {oop(a 'java/util/concurrent/atomic/LongAdder'{0x00000006b5bd29a0})}  0x00007feb1c160a10: mov 0x18(%r10),%r9d ;*getfield cells {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.LongAdder::add@1 (line 87)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a14: mov 0xc(%r12,%r9,8),%r11d ; implicit exception: dispatches to 0x00007feb1c160d44  ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c160a19: mov 0x390(%r15),%r10  0x00007feb1c160a20: mov (%r10),%r10 ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.Striped64::getProbe@3 (line 195)  ; - java.util.concurrent.atomic.LongAdder::add@27 (line 88)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a23: mov 0xc(%r10),%ecx ;*invokevirtual getInt {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.invoke.VarHandleInts$FieldInstanceReadOnly::get@23 (line 90)  ; - java.lang.invoke.VarHandleGuards::guard_L_I@42 (line 109)  ; - java.util.concurrent.atomic.Striped64::getProbe@6 (line 195)  ; - java.util.concurrent.atomic.LongAdder::add@27 (line 88)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a27: lea -0x1(%r11),%r8d  0x00007feb1c160a2b: test %r8d,%r8d  0x00007feb1c160a2e: jl 0x00007feb1c160bc8 ;*iflt {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.LongAdder::add@46 (line 90)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a34: mov %ecx,%ebp  0x00007feb1c160a36: and %r8d,%ebp ;*iand {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.LongAdder::add@54 (line 90)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a39: test %r11d,%r11d  0x00007feb1c160a3c: jbe 0x00007feb1c160bac  0x00007feb1c160a42: lea (%r12,%r9,8),%r10  0x00007feb1c160a46: mov 0x10(%r10,%rbp,4),%ebp ;*aaload {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.LongAdder::add@55 (line 90)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a4b: mov 0x90(%r12,%rbp,8),%rax ; implicit exception: dispatches to 0x00007feb1c160d60  0x00007feb1c160a53: lea -0x1(%rax),%r10  0x00007feb1c160a57: lock cmpxchg %r10,0x90(%r12,%rbp,8)  0x00007feb1c160a61: sete %bpl  0x00007feb1c160a65: movzbl %bpl,%ebp ;*invokevirtual weakCompareAndSetLongRelease {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.invoke.VarHandleLongs$FieldInstanceReadWrite::weakCompareAndSetRelease@29 (line 241)  ; - java.lang.invoke.VarHandleGuards::guard_LJJ_Z@51 (line 204)  ; - java.util.concurrent.atomic.Striped64$Cell::cas@6 (line 130)  ; - java.util.concurrent.atomic.LongAdder::add@76 (line 92)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a69: test %ebp,%ebp  0x00007feb1c160a6b: je 0x00007feb1c160c84 ;*ifne {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.LongAdder::add@82 (line 92)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a71: mov (%rsp),%r10  0x00007feb1c160a75: mov %r12d,0x4c(%r10) ;*putfield inheritableThreadLocals {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::clearReferences@7 (line 1469)  ; - java.lang.VirtualThread::afterDone@76 (line 679)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a79: mov 0x64(%r10),%ebp  0x00007feb1c160a7d: mov %r12d,0x48(%r10) ;*getfield uncaughtExceptionHandler {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::clearReferences@11 (line 1470)  ; - java.lang.VirtualThread::afterDone@76 (line 679)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a81: test %ebp,%ebp  0x00007feb1c160a83: jne 0x00007feb1c160c9c ;*ifnull {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::clearReferences@14 (line 1470)  ; - java.lang.VirtualThread::afterDone@76 (line 679)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a89: mov 0x5c(%r10),%ebp ;*getfield nioBlocker {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::clearReferences@23 (line 1472)  ; - java.lang.VirtualThread::afterDone@76 (line 679)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a8d: test %ebp,%ebp  0x00007feb1c160a8f: jne 0x00007feb1c160cb0 ;*ifnull {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::clearReferences@26 (line 1472)  ; - java.lang.VirtualThread::afterDone@76 (line 679)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  0x00007feb1c160a95: add $0x40,%rsp  0x00007feb1c160a99: pop %rbp  0x00007feb1c160a9a: cmp 0x4e0(%r15),%rsp  0x00007feb1c160aa1: jb 0x00007feb1c160ae4  0x00007feb1c160aa7: mov %r15,%rdi  0x00007feb1c160aaa: test $0xf,%spl  0x00007feb1c160aae: je 0x00007feb1c160ac6  0x00007feb1c160ab4: sub $0x8,%rsp  0x00007feb1c160ab8: call 0x00007feb32c10ae0 ; {runtime_call SharedRuntime::enable_stack_reserved_zone(JavaThread*)}  0x00007feb1c160abd: add $0x8,%rsp  0x00007feb1c160ac1: jmp 0x00007feb1c160acb  0x00007feb1c160ac6: call 0x00007feb32c10ae0 ; {runtime_call SharedRuntime::enable_stack_reserved_zone(JavaThread*)}  0x00007feb1c160acb: jmp 0x00007feb1bb85360 ; {runtime_call Shared Runtime throw_delayed_StackOverflowError_blob}  0x00007feb1c160ad0: movabs $0x7feb32e9ef41,%rdi ; {external_word}  0x00007feb1c160ada: and $0xfffffffffffffff0,%rsp  0x00007feb1c160ade: call 0x00007feb32a35120 ; {runtime_call MacroAssembler::debug64(char*, long, long*)}  0x00007feb1c160ae3: hlt   0x00007feb1c160ae4: cmp 0x28(%r15),%rsp ; {poll_return}  0x00007feb1c160ae8: ja 0x00007feb1c160d78  0x00007feb1c160aee: ret ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c160aef: mov %r12b,0x314(%r15) ;*invokevirtual setCurrentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@31 (line 523)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  0x00007feb1c160af6: jmp 0x00007feb1c160869 ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c160afb: mov %r12b,0x314(%r15) ;*invokevirtual setCurrentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@65 (line 509)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  0x00007feb1c160b02: jmp 0x00007feb1c1607c0  0x00007feb1c160b07: mov $0xfffffff6,%esi  0x00007feb1c160b0c: mov (%rsp),%rbp  0x00007feb1c160b10: data16 xchg %ax,%ax  0x00007feb1c160b13: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=Oop }  ;*invokevirtual run {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@100 (line 303)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160b18: nopl 0x1000610(%rax,%rax,1) ;*invokevirtual run {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@100 (line 303)  ; {post_call_nop}  0x00007feb1c160b20: mov $0xfffffff6,%esi  0x00007feb1c160b25: mov (%rsp),%rbp  0x00007feb1c160b29: xchg %ax,%ax  0x00007feb1c160b2b: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=Oop [24]=NarrowOop }  ;*monitorenter {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@40 (line 526)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160b30: nopl 0x2000628(%rax,%rax,1) ;*monitorenter {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@40 (line 526)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  ; {post_call_nop}  0x00007feb1c160b38: mov $0xffffffde,%esi  0x00007feb1c160b3d: mov (%rsp),%rbp  0x00007feb1c160b41: mov %r8,0x10(%rsp)  0x00007feb1c160b46: nop  0x00007feb1c160b47: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=Oop [16]=Oop }  ;*invokevirtual isInterrupted {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@29 (line 499)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160b4c: nopl 0x3000644(%rax,%rax,1) ;*invokevirtual isInterrupted {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@29 (line 499)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  ; {post_call_nop}  0x00007feb1c160b54: mov %rbp,%rsi  0x00007feb1c160b57: lea 0x30(%rsp),%rdx  0x00007feb1c160b5c: mov %r8,0x20(%rsp)  0x00007feb1c160b61: xchg %ax,%ax  0x00007feb1c160b63: call 0x00007feb1bc465e0 ; ImmutableOopMap {rbp=Oop [0]=Oop [24]=NarrowOop [32]=Oop }  ;*monitorenter {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@40 (line 526)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  ; {runtime_call C2 Runtime complete_monitor_locking}  0x00007feb1c160b68: nopl 0x4000660(%rax,%rax,1) ; {post_call_nop}  0x00007feb1c160b70: jmp 0x00007feb1c1608f6  0x00007feb1c160b75: mov %rbp,%rdi  0x00007feb1c160b78: lea 0x30(%rsp),%rsi ;*invokestatic currentThread {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@0 (line 275)  0x00007feb1c160b7d: mov %r15,%rdx  0x00007feb1c160b80: movabs $0x7feb32c0f1c0,%r10  0x00007feb1c160b8a: call *%r10  0x00007feb1c160b8d: nopl 0x0(%rax,%rax,1) ;*monitorexit {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@47 (line 528)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  ; {post_call_nop}  0x00007feb1c160b95: jmp 0x00007feb1c1609a4  0x00007feb1c160b9a: mov $0xffffffde,%esi  0x00007feb1c160b9f: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=NarrowOop [0]=Oop }  ;*invokevirtual onExit {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160ba4: nopl 0x500069c(%rax,%rax,1) ;*invokevirtual onExit {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {post_call_nop}  0x00007feb1c160bac: mov $0xffffffe4,%esi  0x00007feb1c160bb1: mov %ecx,0x8(%rsp)  0x00007feb1c160bb5: mov %r9d,0xc(%rsp)  0x00007feb1c160bba: nop  0x00007feb1c160bbb: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {[0]=Oop [12]=NarrowOop }  ;*aaload {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.LongAdder::add@55 (line 90)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160bc0: nopl 0x60006b8(%rax,%rax,1) ;*aaload {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.LongAdder::add@55 (line 90)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {post_call_nop}  0x00007feb1c160bc8: mov $0xffffff4d,%esi  0x00007feb1c160bcd: mov (%rsp),%rbp  0x00007feb1c160bd1: mov %ecx,(%rsp)  0x00007feb1c160bd4: mov %r8d,0x4(%rsp)  0x00007feb1c160bd9: xchg %ax,%ax  0x00007feb1c160bdb: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=Oop }  ;*iflt {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.util.concurrent.atomic.LongAdder::add@46 (line 90)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160be0: nopl 0x70006d8(%rax,%rax,1) ;*iflt {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.LongAdder::add@46 (line 90)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {post_call_nop}  0x00007feb1c160be8: mov $0xffffff4d,%esi  0x00007feb1c160bed: mov (%rsp),%rbp  0x00007feb1c160bf1: mov %r11d,0x4(%rsp)  0x00007feb1c160bf6: nop  0x00007feb1c160bf7: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=Oop }  ;*if_icmpeq {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.lang.VirtualThread::runContinuation@24 (line 281)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160bfc: nopl 0x80006f4(%rax,%rax,1) ;*if_icmpeq {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@24 (line 281)  ; {post_call_nop}  0x00007feb1c160c04: mov $0xffffff4d,%esi  0x00007feb1c160c09: xchg %ax,%ax  0x00007feb1c160c0b: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {}  ;*ifne {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.lang.VirtualThread::runContinuation@51 (line 284)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160c10: nopl 0x9000708(%rax,%rax,1) ;*ifne {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@51 (line 284)  ; {post_call_nop}  0x00007feb1c160c18: mov $0xffffff4d,%esi  0x00007feb1c160c1d: mov %r8,0x10(%rsp)  0x00007feb1c160c22: nop  0x00007feb1c160c23: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {[0]=Oop [16]=Oop }  ;*ifeq {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.lang.VirtualThread::mount@18 (line 497)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160c28: nopl 0xa000720(%rax,%rax,1) ;*ifeq {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@18 (line 497)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  ; {post_call_nop}  0x00007feb1c160c30: mov $0xffffff4d,%esi  0x00007feb1c160c35: mov %r8,0x10(%rsp)  0x00007feb1c160c3a: nop  0x00007feb1c160c3b: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {[0]=Oop [16]=Oop }  ;*ifeq {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.lang.VirtualThread::mount@32 (line 499)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160c40: nopl 0xb000738(%rax,%rax,1) ;*ifeq {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::mount@32 (line 499)  ; - java.lang.VirtualThread::runContinuation@93 (line 301)  ; {post_call_nop}  0x00007feb1c160c48: mov $0xffffff4d,%esi  0x00007feb1c160c4d: xchg %ax,%ax  0x00007feb1c160c4f: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {[0]=Oop [24]=NarrowOop }  ;*ifeq {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.lang.Thread::clearInterrupt@4 (line 1623)  ; - java.lang.VirtualThread::unmount@57 (line 529)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160c54: nopl 0xc00074c(%rax,%rax,1) ;*ifeq {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::clearInterrupt@4 (line 1623)  ; - java.lang.VirtualThread::unmount@57 (line 529)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  ; {post_call_nop}  0x00007feb1c160c5c: mov $0xffffff4d,%esi  0x00007feb1c160c61: xchg %ax,%ax  0x00007feb1c160c63: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {[0]=Oop }  ;*ifeq {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.lang.VirtualThread::runContinuation@114 (line 306)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160c68: nopl 0xd000760(%rax,%rax,1) ;*ifeq {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@114 (line 306)  ; {post_call_nop}  0x00007feb1c160c70: mov $0xffffff4d,%esi  0x00007feb1c160c75: xchg %ax,%ax  0x00007feb1c160c77: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=NarrowOop [0]=Oop }  ;*ifnull {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.lang.VirtualThread::afterDone@33 (line 668)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160c7c: nopl 0xe000774(%rax,%rax,1) ;*ifnull {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::afterDone@33 (line 668)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {post_call_nop}  0x00007feb1c160c84: mov $0xffffff4d,%esi  0x00007feb1c160c89: mov %ecx,0x8(%rsp)  0x00007feb1c160c8d: xchg %ax,%ax  0x00007feb1c160c8f: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {[0]=Oop }  ;*ifne {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.util.concurrent.atomic.LongAdder::add@82 (line 92)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160c94: nopl 0xf00078c(%rax,%rax,1) ;*ifne {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.LongAdder::add@82 (line 92)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {post_call_nop}  0x00007feb1c160c9c: mov $0xffffff4d,%esi  0x00007feb1c160ca1: xchg %ax,%ax  0x00007feb1c160ca3: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=NarrowOop [0]=Oop }  ;*ifnull {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.lang.Thread::clearReferences@14 (line 1470)  ; - java.lang.VirtualThread::afterDone@76 (line 679)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160ca8: nopl 0x100007a0(%rax,%rax,1) ;*ifnull {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::clearReferences@14 (line 1470)  ; - java.lang.VirtualThread::afterDone@76 (line 679)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {post_call_nop}  0x00007feb1c160cb0: mov $0xffffff4d,%esi  0x00007feb1c160cb5: xchg %ax,%ax  0x00007feb1c160cb7: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=NarrowOop [0]=Oop }  ;*ifnull {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.lang.Thread::clearReferences@26 (line 1472)  ; - java.lang.VirtualThread::afterDone@76 (line 679)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160cbc: nopl 0x110007b4(%rax,%rax,1) ;*ifnull {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::clearReferences@26 (line 1472)  ; - java.lang.VirtualThread::afterDone@76 (line 679)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {post_call_nop}  0x00007feb1c160cc4: mov $0xffffff66,%esi  0x00007feb1c160cc9: mov (%rsp),%rbp  0x00007feb1c160ccd: mov %r10,(%rsp)  0x00007feb1c160cd1: xchg %ax,%ax  0x00007feb1c160cd3: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=Oop [0]=Oop }  ;*instanceof {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.Thread::isVirtual@1 (line 1373)  ; - java.lang.VirtualThread::runContinuation@3 (line 275)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160cd8: nopl 0x120007d0(%rax,%rax,1) ;*invokevirtual run {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@100 (line 303)  ; {post_call_nop}  0x00007feb1c160ce0: mov $0xffffff9d,%esi  0x00007feb1c160ce5: mov (%rsp),%rbp  0x00007feb1c160ce9: mov %rax,(%rsp)  0x00007feb1c160ced: xchg %ax,%ax  0x00007feb1c160cef: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=Oop [0]=Oop }  ;*astore_2 {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@131 (line 305)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160cf4: nopl 0x130007ec(%rax,%rax,1) ; {post_call_nop}  0x00007feb1c160cfc: mov $0xfffffff6,%esi  0x00007feb1c160d01: mov (%rsp),%rbp  0x00007feb1c160d05: xchg %ax,%ax  0x00007feb1c160d07: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=Oop }  ;*invokevirtual clearInterrupt {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@57 (line 529)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160d0c: nopl 0x14000804(%rax,%rax,1) ;*invokevirtual clearInterrupt {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::unmount@57 (line 529)  ; - java.lang.VirtualThread::runContinuation@104 (line 305)  ; {post_call_nop}  0x00007feb1c160d14: mov $0xfffffff6,%esi  0x00007feb1c160d19: mov %r10,%rbp  0x00007feb1c160d1c: data16 xchg %ax,%ax  0x00007feb1c160d1f: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=Oop }  ;*invokevirtual isDone {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@111 (line 306)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160d24: nopl 0x1500081c(%rax,%rax,1) ;*invokevirtual isDone {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::runContinuation@111 (line 306)  ; {post_call_nop}  0x00007feb1c160d2c: mov $0xfffffff6,%esi  0x00007feb1c160d31: mov %r10,%rbp  0x00007feb1c160d34: data16 xchg %ax,%ax  0x00007feb1c160d37: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=Oop }  ;*invokevirtual onExit {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160d3c: nopl 0x16000834(%rax,%rax,1) ;*invokevirtual onExit {reexecute=0 rethrow=0 return_oop=0}  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {post_call_nop}  0x00007feb1c160d44: mov $0xffffff4d,%esi  0x00007feb1c160d49: mov (%rsp),%rbp  0x00007feb1c160d4d: mov %r9d,0x4(%rsp)  0x00007feb1c160d52: nop  0x00007feb1c160d53: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=Oop [4]=NarrowOop }  ;*ifnonnull {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.util.concurrent.atomic.LongAdder::add@6 (line 87)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160d58: nopl 0x17000850(%rax,%rax,1) ;*ifnonnull {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.LongAdder::add@6 (line 87)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {post_call_nop}  0x00007feb1c160d60: mov $0xffffff4d,%esi  0x00007feb1c160d65: mov %ecx,0x8(%rsp)  0x00007feb1c160d69: xchg %ax,%ax  0x00007feb1c160d6b: call 0x00007feb1bc3e5e0 ; ImmutableOopMap {rbp=NarrowOop [0]=Oop }  ;*ifnull {reexecute=1 rethrow=0 return_oop=0}  ; - (reexecute) java.util.concurrent.atomic.LongAdder::add@59 (line 90)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {runtime_call UncommonTrapBlob}  0x00007feb1c160d70: nopl 0x18000868(%rax,%rax,1) ;*ifnull {reexecute=0 rethrow=0 return_oop=0}  ; - java.util.concurrent.atomic.LongAdder::add@59 (line 90)  ; - jdk.internal.vm.ThreadContainers$RootContainer$CountingRootContainer::onExit@27 (line 280)  ; - java.lang.VirtualThread::afterDone@72 (line 675)  ; - java.lang.VirtualThread::afterDone@2 (line 653)  ; - java.lang.VirtualThread::runContinuation@118 (line 307)  ; {post_call_nop}  0x00007feb1c160d78: movabs $0x7feb1c160ae4,%r10 ; {internal_word}  0x00007feb1c160d82: mov %r10,0x460(%r15)  0x00007feb1c160d89: jmp 0x00007feb1bb8e060 ; {runtime_call SafepointBlob}  0x00007feb1c160d8e: addl $0x8,0x6f8(%r15)  0x00007feb1c160d96: jmp 0x00007feb1c16099e  0x00007feb1c160d9b: call Stub::nmethod_entry_barrier ; {runtime_call StubRoutines (final stubs)}  0x00007feb1c160da0: jmp 0x00007feb1c1606aa  0x00007feb1c160da5: hlt   0x00007feb1c160da6: hlt   0x00007feb1c160da7: hlt  [Stub Code]  0x00007feb1c160da8: movabs $0x0,%rbx ; {no_reloc}  0x00007feb1c160db2: jmp 0x00007feb1c160db2 ; {runtime_call nmethod} [Exception Handler]  0x00007feb1c160db7: jmp 0x00007feb1bc3e8e0 ; {runtime_call ExceptionBlob} [Deopt Handler Code]  0x00007feb1c160dbc: call 0x00007feb1c160dc1  0x00007feb1c160dc1: subq $0x5,(%rsp)  0x00007feb1c160dc6: jmp 0x00007feb1bb8c900 ; {runtime_call DeoptimizationBlob}  0x00007feb1c160dcb: hlt   0x00007feb1c160dcc: hlt   0x00007feb1c160dcd: hlt   0x00007feb1c160dce: hlt   0x00007feb1c160dcf: hlt  -------------------------------------------------------------------------------- [/Disassembly] 

[franz1981]

Which atomic ops are

 0x00007feb1c1606e5: lock cmpxchg %r10d,0x80(%rsi) ;*invokevirtual compareAndSetInt {reexecute=0 rethrow=0 return_oop=0} ; - java.lang.VirtualThread::compareAndSetState@9 (line 1358) ; - java.lang.VirtualThread::runContinuation@48 (line 284) 

which is related to https://github.com/openjdk/jdk/blob/jdk-24%2B36/src/java.base/share/classes/java/lang/VirtualThread.java#L1358

 0x00007feb1c160727: mov %r12b,(%r11,%r10,1) 0x00007feb1c16072b: lock addl $0x0,-0x40(%rsp) ;*putfield carrierThread {reexecute=0 rethrow=0 return_oop=0} 

which is related to https://github.com/openjdk/jdk/blob/jdk-24%2B36/src/java.base/share/classes/java/lang/VirtualThread.java#L1381

 0x00007feb1c1609d0: movl $0x63,0x80(%r10) ; {no_reloc} 0x00007feb1c1609db: lock addl $0x0,-0x40(%rsp) ;*putfield state {reexecute=0 rethrow=0 return_oop=0} ; - java.lang.VirtualThread::setState@2 (line 1354) ; - java.lang.VirtualThread::afterDone@24 (line 664) ; - java.lang.VirtualThread::afterDone@2 (line 653) ; - java.lang.VirtualThread::runContinuation@118 (line 307) 

which is related to https://github.com/openjdk/jdk/blob/jdk-24%2B36/src/java.base/share/classes/java/lang/VirtualThread.java#L1354

and others...

This can be reproduced via franz1981/java-puzzles@7ae7468 quite easily

[franz1981]

FYI plaintext on my machine (with fewer connections/threads etc etc) reports
with event loop carrier

 Thread Stats Avg Stdev Max +/- Stdev Latency 144.50us 2.09ms 78.53ms 82.03% Req/Sec 0.94M 127.60k 1.22M 81.17% Latency Distribution 50% 59.00us 75% 88.00us 90% 0.00us 99% 0.00us 74938880 requests in 40.10s, 9.14GB read Requests/sec: 1868835.89 Transfer/sec: 233.48MB 

without event loop carrier:

 Thread Stats Avg Stdev Max +/- Stdev Latency 150.18us 2.16ms 79.88ms 78.56% Req/Sec 1.12M 152.01k 1.34M 86.16% Latency Distribution 50% 52.00us 75% 90.00us 90% 0.00us 99% 0.00us 88888048 requests in 40.10s, 10.84GB read Requests/sec: 2216695.70 Transfer/sec: 276.93MB 

which is indeed a clear signal of the overhead of allocating virtual threads with pipelining (i.e. in tight loops)

[franz1981]

It looks like the json test is not right with loom, by addding -Djdk.traceVirtualThreadLocals=true

VirtualThread[#8353]/runnable@multithreadVirtualEventExecutorGroup-2-1 java.base/java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:235) java.base/java.lang.ThreadLocal.get(ThreadLocal.java:193) java.base/java.lang.ThreadLocal.get(ThreadLocal.java:171) com.jsoniter.spi.JsoniterSpi.getCurrentConfig(JsoniterSpi.java:51) com.jsoniter.output.JsonStream.currentConfig(JsonStream.java:389) com.jsoniter.output.JsonStream.writeVal(JsonStream.java:379) hello.JsonUtils.serializeMsg(JsonUtils.java:65) hello.HttpResponses.makeJsonResponse(HttpResponses.java:33) hello.loom.VirtualThreadHelloServerHandler.lambda$writeJsonResponse$0(VirtualThreadHelloServerHandler.java:34) java.base/java.lang.VirtualThread.run(VirtualThread.java:466) 

and that means that is not a good idea to use this json library, if not with some small adjustment.

And https://github.com/openjdk/jdk/blob/jdk-24%2B36/src/java.base/share/classes/jdk/internal/vm/ThreadContainers.java#L60

it looks like that disabling tracking virtual threads still hit a LongAdder usage in the hot path .-.

Flamegraph for the JSON endpoint:

netty_json.zip

with event loop as a carrier:

 Thread Stats Avg Stdev Max +/- Stdev Latency 102.94us 1.81ms 73.37ms 99.79% Req/Sec 145.20k 16.91k 168.85k 88.03% Latency Distribution 50% 30.00us 75% 35.00us 90% 39.00us 99% 133.00us 11570061 requests in 40.10s, 1.63GB read Requests/sec: 288530.65 Transfer/sec: 41.55MB 

and without (i.e. no loom):

 Thread Stats Avg Stdev Max +/- Stdev Latency 102.44us 1.84ms 74.03ms 99.80% Req/Sec 144.96k 16.89k 170.26k 86.78% Latency Distribution 50% 30.00us 75% 34.00us 90% 38.00us 99% 111.00us 11552394 requests in 40.10s, 1.62GB read Requests/sec: 288093.97 Transfer/sec: 41.49MB 

[franz1981]

@NateBrady23 ready to go ;)

[msmith-techempower]

I am now @NateBrady23 - LGTM!