Add span propagation for Pekko scheduled tasks #8765

nikita-tkachenko-datadog · 2025-05-05T10:30:46Z

What Does This Do

Implements propagation of span context for Pekko's asynchronous scheduled tasks.
The new instrumentation plugs into LightArrayRevolverScheduler#schedule method: when a Runnable is scheduled, the context of currently active span is captured and attached to that Runnable.
Later the existing runnable instrumentation activates this context when the task is executed (potentially in a different thread).

Activating a continuation of a span is one-time only, meaning that a context that was captured can only be activated once. For this reason this specific method is instrumented, because it works by re-scheduling periodic tasks after they finish executing (so a new continuation is captured every time a periodic task is triggered).

Motivation

CI Visibility's per-test code coverage relies on spans propagation to capture coverage data from different threads.
Without this instrumentation coverage data for test cases that use Pekko's scheduled tasks is incomplete.

Additional Notes

For now the new instrumentation is disabled by default and has to be enabled explicitly with a config property.
This is done to reduce possible negative impact.

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the public documentation in case of new configuration flag or behavior

Jira ticket: SDTEST-1957

pr-commenter · 2025-05-05T11:20:13Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/pekko-scheduler-instrumentation
git_commit_date	1746516330	1746520049
git_commit_sha	`0e18e0e`	`2ce5ed4`
release_version	1.49.0-SNAPSHOT~0e18e0e887	1.49.0-SNAPSHOT~2ce5ed4064

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1746523418	1746523418
ci_job_id	925006416	925006416
ci_pipeline_id	64211741	64211741
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-drymq1by-project-304-concurrent-0-18is7160 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-drymq1by-project-304-concurrent-0-18is7160 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 65 metrics, 6 unstable metrics.

Startup time reports for petclinic

gantt title petclinic - global startup overhead: candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887 dateFormat X axisFormat %s section tracing Agent [baseline] (1.007 s) : 0, 1007163 Total [baseline] (10.516 s) : 0, 10516407 Agent [candidate] (1.006 s) : 0, 1005886 Total [candidate] (10.519 s) : 0, 10518843 section appsec Agent [baseline] (1.151 s) : 0, 1151272 Total [baseline] (10.71 s) : 0, 10709847 Agent [candidate] (1.155 s) : 0, 1155321 Total [candidate] (10.685 s) : 0, 10684596 section iast Agent [baseline] (1.137 s) : 0, 1137343 Total [baseline] (10.799 s) : 0, 10798853 Agent [candidate] (1.137 s) : 0, 1136719 Total [candidate] (10.834 s) : 0, 10834103 section profiling Agent [baseline] (1.261 s) : 0, 1260746 Total [baseline] (10.856 s) : 0, 10855576 Agent [candidate] (1.254 s) : 0, 1254370 Total [candidate] (10.752 s) : 0, 10751706

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.007 s	-
Agent	appsec	1.151 s	144.108 ms (14.3%)
Agent	iast	1.137 s	130.179 ms (12.9%)
Agent	profiling	1.261 s	253.583 ms (25.2%)
Total	tracing	10.516 s	-
Total	appsec	10.71 s	193.44 ms (1.8%)
Total	iast	10.799 s	282.445 ms (2.7%)
Total	profiling	10.856 s	339.169 ms (3.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.006 s	-
Agent	appsec	1.155 s	149.435 ms (14.9%)
Agent	iast	1.137 s	130.833 ms (13.0%)
Agent	profiling	1.254 s	248.484 ms (24.7%)
Total	tracing	10.519 s	-
Total	appsec	10.685 s	165.753 ms (1.6%)
Total	iast	10.834 s	315.26 ms (3.0%)
Total	profiling	10.752 s	232.863 ms (2.2%)

gantt title petclinic - break down per module: candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887 dateFormat X axisFormat %s section tracing BytebuddyAgent [baseline] (670.513 ms) : 0, 670513 BytebuddyAgent [candidate] (670.837 ms) : 0, 670837 GlobalTracer [baseline] (240.232 ms) : 0, 240232 GlobalTracer [candidate] (239.564 ms) : 0, 239564 AppSec [baseline] (54.77 ms) : 0, 54770 AppSec [candidate] (54.515 ms) : 0, 54515 Debugger [baseline] (6.146 ms) : 0, 6146 Debugger [candidate] (6.149 ms) : 0, 6149 Remote Config [baseline] (694.791 µs) : 0, 695 Remote Config [candidate] (704.954 µs) : 0, 705 Telemetry [baseline] (11.413 ms) : 0, 11413 Telemetry [candidate] (10.657 ms) : 0, 10657 section appsec BytebuddyAgent [baseline] (690.224 ms) : 0, 690224 BytebuddyAgent [candidate] (693.26 ms) : 0, 693260 GlobalTracer [baseline] (236.687 ms) : 0, 236687 GlobalTracer [candidate] (237.845 ms) : 0, 237845 AppSec [baseline] (175.19 ms) : 0, 175190 AppSec [candidate] (175.358 ms) : 0, 175358 Debugger [baseline] (5.889 ms) : 0, 5889 Debugger [candidate] (5.882 ms) : 0, 5882 Remote Config [baseline] (643.278 µs) : 0, 643 Remote Config [candidate] (630.785 µs) : 0, 631 Telemetry [baseline] (8.151 ms) : 0, 8151 Telemetry [candidate] (7.78 ms) : 0, 7780 IAST [baseline] (21.854 ms) : 0, 21854 IAST [candidate] (22.017 ms) : 0, 22017 section iast BytebuddyAgent [baseline] (790.015 ms) : 0, 790015 BytebuddyAgent [candidate] (788.986 ms) : 0, 788986 GlobalTracer [baseline] (230.359 ms) : 0, 230359 GlobalTracer [candidate] (230.484 ms) : 0, 230484 AppSec [baseline] (56.202 ms) : 0, 56202 AppSec [candidate] (56.559 ms) : 0, 56559 Debugger [baseline] (5.92 ms) : 0, 5920 Debugger [candidate] (5.971 ms) : 0, 5971 Remote Config [baseline] (589.248 µs) : 0, 589 Remote Config [candidate] (596.381 µs) : 0, 596 Telemetry [baseline] (7.968 ms) : 0, 7968 Telemetry [candidate] (7.971 ms) : 0, 7971 IAST [baseline] (22.824 ms) : 0, 22824 IAST [candidate] (22.726 ms) : 0, 22726 section profiling BytebuddyAgent [baseline] (662.136 ms) : 0, 662136 BytebuddyAgent [candidate] (661.115 ms) : 0, 661115 GlobalTracer [baseline] (380.227 ms) : 0, 380227 GlobalTracer [candidate] (378.28 ms) : 0, 378280 AppSec [baseline] (55.296 ms) : 0, 55296 AppSec [candidate] (53.488 ms) : 0, 53488 Debugger [baseline] (6.184 ms) : 0, 6184 Debugger [candidate] (6.11 ms) : 0, 6110 Remote Config [baseline] (669.768 µs) : 0, 670 Remote Config [candidate] (650.246 µs) : 0, 650 Telemetry [baseline] (8.29 ms) : 0, 8290 Telemetry [candidate] (8.115 ms) : 0, 8115 ProfilingAgent [baseline] (97.649 ms) : 0, 97649 ProfilingAgent [candidate] (96.452 ms) : 0, 96452 Profiling [baseline] (97.674 ms) : 0, 97674 Profiling [candidate] (96.475 ms) : 0, 96475

Startup time reports for insecure-bank

gantt title insecure-bank - global startup overhead: candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887 dateFormat X axisFormat %s section tracing Agent [baseline] (1.016 s) : 0, 1016350 Total [baseline] (8.665 s) : 0, 8665379 Agent [candidate] (1.003 s) : 0, 1003472 Total [candidate] (8.604 s) : 0, 8603955 section iast Agent [baseline] (1.145 s) : 0, 1144953 Total [baseline] (9.224 s) : 0, 9223889 Agent [candidate] (1.136 s) : 0, 1136445 Total [candidate] (9.209 s) : 0, 9208890 section iast_HARDCODED_SECRET_DISABLED Agent [baseline] (1.136 s) : 0, 1135993 Total [baseline] (9.168 s) : 0, 9167617 Agent [candidate] (1.139 s) : 0, 1139334 Total [candidate] (9.159 s) : 0, 9159491 section iast_TELEMETRY_OFF Agent [baseline] (1.131 s) : 0, 1130890 Total [baseline] (9.182 s) : 0, 9181618 Agent [candidate] (1.146 s) : 0, 1145755 Total [candidate] (9.223 s) : 0, 9223204

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.016 s	-
Agent	iast	1.145 s	128.603 ms (12.7%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.136 s	119.643 ms (11.8%)
Agent	iast_TELEMETRY_OFF	1.131 s	114.539 ms (11.3%)
Total	tracing	8.665 s	-
Total	iast	9.224 s	558.51 ms (6.4%)
Total	iast_HARDCODED_SECRET_DISABLED	9.168 s	502.238 ms (5.8%)
Total	iast_TELEMETRY_OFF	9.182 s	516.24 ms (6.0%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.003 s	-
Agent	iast	1.136 s	132.973 ms (13.3%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.139 s	135.861 ms (13.5%)
Agent	iast_TELEMETRY_OFF	1.146 s	142.283 ms (14.2%)
Total	tracing	8.604 s	-
Total	iast	9.209 s	604.936 ms (7.0%)
Total	iast_HARDCODED_SECRET_DISABLED	9.159 s	555.536 ms (6.5%)
Total	iast_TELEMETRY_OFF	9.223 s	619.25 ms (7.2%)

gantt title insecure-bank - break down per module: candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887 dateFormat X axisFormat %s section tracing BytebuddyAgent [baseline] (674.954 ms) : 0, 674954 BytebuddyAgent [candidate] (668.362 ms) : 0, 668362 GlobalTracer [baseline] (241.086 ms) : 0, 241086 GlobalTracer [candidate] (238.712 ms) : 0, 238712 AppSec [baseline] (54.85 ms) : 0, 54850 AppSec [candidate] (54.287 ms) : 0, 54287 Debugger [baseline] (8.39 ms) : 0, 8390 Debugger [candidate] (8.254 ms) : 0, 8254 Remote Config [baseline] (708.713 µs) : 0, 709 Remote Config [candidate] (684.968 µs) : 0, 685 Telemetry [baseline] (12.71 ms) : 0, 12710 Telemetry [candidate] (9.655 ms) : 0, 9655 section iast BytebuddyAgent [baseline] (795.414 ms) : 0, 795414 BytebuddyAgent [candidate] (788.98 ms) : 0, 788980 GlobalTracer [baseline] (231.577 ms) : 0, 231577 GlobalTracer [candidate] (230.401 ms) : 0, 230401 AppSec [baseline] (56.718 ms) : 0, 56718 AppSec [candidate] (56.454 ms) : 0, 56454 Debugger [baseline] (6.027 ms) : 0, 6027 Debugger [candidate] (5.921 ms) : 0, 5921 Remote Config [baseline] (608.962 µs) : 0, 609 Remote Config [candidate] (610.919 µs) : 0, 611 Telemetry [baseline] (8.079 ms) : 0, 8079 Telemetry [candidate] (7.884 ms) : 0, 7884 IAST [baseline] (22.928 ms) : 0, 22928 IAST [candidate] (22.752 ms) : 0, 22752 section iast_HARDCODED_SECRET_DISABLED BytebuddyAgent [baseline] (789.092 ms) : 0, 789092 BytebuddyAgent [candidate] (792.014 ms) : 0, 792014 GlobalTracer [baseline] (229.824 ms) : 0, 229824 GlobalTracer [candidate] (230.345 ms) : 0, 230345 AppSec [baseline] (56.416 ms) : 0, 56416 AppSec [candidate] (56.361 ms) : 0, 56361 Debugger [baseline] (5.935 ms) : 0, 5935 Debugger [candidate] (5.872 ms) : 0, 5872 Remote Config [baseline] (582.916 µs) : 0, 583 Remote Config [candidate] (589.792 µs) : 0, 590 Telemetry [baseline] (7.887 ms) : 0, 7887 Telemetry [candidate] (7.882 ms) : 0, 7882 IAST [baseline] (22.837 ms) : 0, 22837 IAST [candidate] (22.824 ms) : 0, 22824 section iast_TELEMETRY_OFF BytebuddyAgent [baseline] (785.117 ms) : 0, 785117 BytebuddyAgent [candidate] (793.821 ms) : 0, 793821 GlobalTracer [baseline] (229.488 ms) : 0, 229488 GlobalTracer [candidate] (233.748 ms) : 0, 233748 AppSec [baseline] (56.408 ms) : 0, 56408 AppSec [candidate] (57.386 ms) : 0, 57386 Debugger [baseline] (5.923 ms) : 0, 5923 Debugger [candidate] (6.016 ms) : 0, 6016 Remote Config [baseline] (584.579 µs) : 0, 585 Remote Config [candidate] (615.97 µs) : 0, 616 Telemetry [baseline] (7.749 ms) : 0, 7749 Telemetry [candidate] (7.783 ms) : 0, 7783 IAST [baseline] (22.288 ms) : 0, 22288 IAST [candidate] (22.849 ms) : 0, 22849

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-05-06T08:54:08	2025-05-06T09:01:52
git_branch	master	nikita-tkachenko/pekko-scheduler-instrumentation
git_commit_date	1746516330	1746520049
git_commit_sha	`0e18e0e`	`2ce5ed4`
release_version	1.49.0-SNAPSHOT~0e18e0e887	1.49.0-SNAPSHOT~2ce5ed4064
start_time	2025-05-06T08:53:54	2025-05-06T09:01:38

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1746522510	1746522510
ci_job_id	925006417	925006417
ci_pipeline_id	64211741	64211741
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-16jikxse-project-304-concurrent-0-jw1942ih 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-16jikxse-project-304-concurrent-0-jw1942ih 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 13 metrics, 17 unstable metrics.

Request duration reports for petclinic

gantt title petclinic - request duration [CI 0.99] : candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887 dateFormat X axisFormat %s section baseline no_agent (1.365 ms) : 1346, 1384 . : milestone, 1365, appsec (1.739 ms) : 1715, 1762 . : milestone, 1739, appsec_no_iast (1.754 ms) : 1731, 1777 . : milestone, 1754, code_origins (1.672 ms) : 1645, 1699 . : milestone, 1672, iast (1.521 ms) : 1497, 1546 . : milestone, 1521, profiling (1.528 ms) : 1504, 1551 . : milestone, 1528, tracing (1.484 ms) : 1459, 1510 . : milestone, 1484, section candidate no_agent (1.365 ms) : 1345, 1384 . : milestone, 1365, appsec (1.751 ms) : 1728, 1775 . : milestone, 1751, appsec_no_iast (1.735 ms) : 1712, 1758 . : milestone, 1735, code_origins (1.688 ms) : 1662, 1715 . : milestone, 1688, iast (1.52 ms) : 1495, 1544 . : milestone, 1520, profiling (1.516 ms) : 1493, 1539 . : milestone, 1516, tracing (1.497 ms) : 1471, 1522 . : milestone, 1497,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.365 ms [1.346 ms, 1.384 ms]	-
appsec	1.739 ms [1.715 ms, 1.762 ms]	373.928 µs (27.4%)
appsec_no_iast	1.754 ms [1.731 ms, 1.777 ms]	389.305 µs (28.5%)
code_origins	1.672 ms [1.645 ms, 1.699 ms]	306.883 µs (22.5%)
iast	1.521 ms [1.497 ms, 1.546 ms]	156.345 µs (11.5%)
profiling	1.528 ms [1.504 ms, 1.551 ms]	163.035 µs (11.9%)
tracing	1.484 ms [1.459 ms, 1.51 ms]	119.626 µs (8.8%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.365 ms [1.345 ms, 1.384 ms]	-
appsec	1.751 ms [1.728 ms, 1.775 ms]	386.43 µs (28.3%)
appsec_no_iast	1.735 ms [1.712 ms, 1.758 ms]	370.566 µs (27.2%)
code_origins	1.688 ms [1.662 ms, 1.715 ms]	323.518 µs (23.7%)
iast	1.52 ms [1.495 ms, 1.544 ms]	154.82 µs (11.3%)
profiling	1.516 ms [1.493 ms, 1.539 ms]	151.314 µs (11.1%)
tracing	1.497 ms [1.471 ms, 1.522 ms]	131.945 µs (9.7%)

Request duration reports for insecure-bank

gantt title insecure-bank - request duration [CI 0.99] : candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887 dateFormat X axisFormat %s section baseline no_agent (386.792 µs) : 367, 407 . : milestone, 387, iast (515.092 µs) : 492, 538 . : milestone, 515, iast_FULL (733.733 µs) : 710, 757 . : milestone, 734, iast_GLOBAL (575.787 µs) : 552, 600 . : milestone, 576, iast_HARDCODED_SECRET_DISABLED (531.206 µs) : 508, 554 . : milestone, 531, iast_INACTIVE (467.789 µs) : 445, 490 . : milestone, 468, iast_TELEMETRY_OFF (515.135 µs) : 492, 538 . : milestone, 515, tracing (470.192 µs) : 448, 492 . : milestone, 470, section candidate no_agent (385.915 µs) : 366, 405 . : milestone, 386, iast (520.042 µs) : 497, 543 . : milestone, 520, iast_FULL (745.264 µs) : 722, 769 . : milestone, 745, iast_GLOBAL (570.204 µs) : 547, 593 . : milestone, 570, iast_HARDCODED_SECRET_DISABLED (526.215 µs) : 503, 549 . : milestone, 526, iast_INACTIVE (470.462 µs) : 449, 492 . : milestone, 470, iast_TELEMETRY_OFF (518.388 µs) : 495, 541 . : milestone, 518, tracing (460.684 µs) : 440, 482 . : milestone, 461,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	386.792 µs [366.81 µs, 406.774 µs]	-
iast	515.092 µs [492.222 µs, 537.962 µs]	128.3 µs (33.2%)
iast_FULL	733.733 µs [710.21 µs, 757.256 µs]	346.941 µs (89.7%)
iast_GLOBAL	575.787 µs [552.037 µs, 599.536 µs]	188.995 µs (48.9%)
iast_HARDCODED_SECRET_DISABLED	531.206 µs [508.484 µs, 553.929 µs]	144.414 µs (37.3%)
iast_INACTIVE	467.789 µs [445.43 µs, 490.149 µs]	80.997 µs (20.9%)
iast_TELEMETRY_OFF	515.135 µs [491.964 µs, 538.305 µs]	128.343 µs (33.2%)
tracing	470.192 µs [448.041 µs, 492.343 µs]	83.4 µs (21.6%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	385.915 µs [366.453 µs, 405.377 µs]	-
iast	520.042 µs [497.175 µs, 542.91 µs]	134.127 µs (34.8%)
iast_FULL	745.264 µs [722.025 µs, 768.504 µs]	359.349 µs (93.1%)
iast_GLOBAL	570.204 µs [547.038 µs, 593.371 µs]	184.289 µs (47.8%)
iast_HARDCODED_SECRET_DISABLED	526.215 µs [503.122 µs, 549.309 µs]	140.3 µs (36.4%)
iast_INACTIVE	470.462 µs [448.949 µs, 491.974 µs]	84.546 µs (21.9%)
iast_TELEMETRY_OFF	518.388 µs [495.302 µs, 541.475 µs]	132.473 µs (34.3%)
tracing	460.684 µs [439.648 µs, 481.719 µs]	74.768 µs (19.4%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/pekko-scheduler-instrumentation
git_commit_date	1746516330	1746520049
git_commit_sha	`0e18e0e`	`2ce5ed4`
release_version	1.49.0-SNAPSHOT~0e18e0e887	1.49.0-SNAPSHOT~2ce5ed4064

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1746523019	1746523019
ci_job_id	925006418	925006418
ci_pipeline_id	64211741	64211741
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-df2tp9k5-project-304-concurrent-0-mntpkoai 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-df2tp9k5-project-304-concurrent-0-mntpkoai 6.8.0-1027-aws #29~22.04.1-Ubuntu SMP Sun Mar 30 07:45:38 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt title tomcat - execution time [CI 0.99] : candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887 dateFormat X axisFormat %s section baseline no_agent (1.469 ms) : 1458, 1481 . : milestone, 1469, appsec (2.405 ms) : 2355, 2455 . : milestone, 2405, iast (2.194 ms) : 2132, 2256 . : milestone, 2194, iast_GLOBAL (2.225 ms) : 2163, 2287 . : milestone, 2225, profiling (2.03 ms) : 1980, 2079 . : milestone, 2030, tracing (2.003 ms) : 1955, 2051 . : milestone, 2003, section candidate no_agent (1.471 ms) : 1460, 1483 . : milestone, 1471, appsec (2.406 ms) : 2356, 2455 . : milestone, 2406, iast (2.184 ms) : 2122, 2246 . : milestone, 2184, iast_GLOBAL (2.219 ms) : 2157, 2282 . : milestone, 2219, profiling (2.482 ms) : 2237, 2726 . : milestone, 2482, tracing (2.002 ms) : 1954, 2050 . : milestone, 2002,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.469 ms [1.458 ms, 1.481 ms]	-
appsec	2.405 ms [2.355 ms, 2.455 ms]	935.728 µs (63.7%)
iast	2.194 ms [2.132 ms, 2.256 ms]	724.545 µs (49.3%)
iast_GLOBAL	2.225 ms [2.163 ms, 2.287 ms]	755.762 µs (51.4%)
profiling	2.03 ms [1.98 ms, 2.079 ms]	560.511 µs (38.2%)
tracing	2.003 ms [1.955 ms, 2.051 ms]	533.599 µs (36.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.471 ms [1.46 ms, 1.483 ms]	-
appsec	2.406 ms [2.356 ms, 2.455 ms]	934.297 µs (63.5%)
iast	2.184 ms [2.122 ms, 2.246 ms]	713.009 µs (48.5%)
iast_GLOBAL	2.219 ms [2.157 ms, 2.282 ms]	748.17 µs (50.9%)
profiling	2.482 ms [2.237 ms, 2.726 ms]	1.01 ms (68.7%)
tracing	2.002 ms [1.954 ms, 2.05 ms]	530.745 µs (36.1%)

Execution time for biojava

gantt title biojava - execution time [CI 0.99] : candidate=1.49.0-SNAPSHOT~2ce5ed4064, baseline=1.49.0-SNAPSHOT~0e18e0e887 dateFormat X axisFormat %s section baseline no_agent (15.28 s) : 15280000, 15280000 . : milestone, 15280000, appsec (15.241 s) : 15241000, 15241000 . : milestone, 15241000, iast (18.803 s) : 18803000, 18803000 . : milestone, 18803000, iast_GLOBAL (18.034 s) : 18034000, 18034000 . : milestone, 18034000, profiling (15.262 s) : 15262000, 15262000 . : milestone, 15262000, tracing (14.937 s) : 14937000, 14937000 . : milestone, 14937000, section candidate no_agent (15.214 s) : 15214000, 15214000 . : milestone, 15214000, appsec (14.99 s) : 14990000, 14990000 . : milestone, 14990000, iast (18.93 s) : 18930000, 18930000 . : milestone, 18930000, iast_GLOBAL (18.246 s) : 18246000, 18246000 . : milestone, 18246000, profiling (15.145 s) : 15145000, 15145000 . : milestone, 15145000, tracing (15.171 s) : 15171000, 15171000 . : milestone, 15171000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.28 s [15.28 s, 15.28 s]	-
appsec	15.241 s [15.241 s, 15.241 s]	-39.0 ms (-0.3%)
iast	18.803 s [18.803 s, 18.803 s]	3.523 s (23.1%)
iast_GLOBAL	18.034 s [18.034 s, 18.034 s]	2.754 s (18.0%)
profiling	15.262 s [15.262 s, 15.262 s]	-18.0 ms (-0.1%)
tracing	14.937 s [14.937 s, 14.937 s]	-343.0 ms (-2.2%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.214 s [15.214 s, 15.214 s]	-
appsec	14.99 s [14.99 s, 14.99 s]	-224.0 ms (-1.5%)
iast	18.93 s [18.93 s, 18.93 s]	3.716 s (24.4%)
iast_GLOBAL	18.246 s [18.246 s, 18.246 s]	3.032 s (19.9%)
profiling	15.145 s [15.145 s, 15.145 s]	-69.0 ms (-0.5%)
tracing	15.171 s [15.171 s, 15.171 s]	-43.0 ms (-0.3%)

amarziali

LGTM. Is it possible to add a specific test for it?

nikita-tkachenko-datadog · 2025-05-05T14:11:38Z

LGTM. Is it possible to add a specific test for it?

Done

…tation

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-logging](https://github.com/googleapis/java-logging) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `3.22.2` -> `3.22.3` | | [com.google.cloud:google-cloud-datastore](https://github.com/googleapis/java-datastore) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.28.0` -> `2.28.1` | | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.48.2` -> `1.49.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | --- ### Release Notes <details> <summary>googleapis/java-logging (com.google.cloud:google-cloud-logging)</summary> ### [`v3.22.3`](https://github.com/googleapis/java-logging/blob/HEAD/CHANGELOG.md#3223-2025-05-06) ##### Bug Fixes - **deps:** Update the Java code generator (gapic-generator-java) to 2.56.3 ([844f4fa](googleapis/java-logging@844f4fa)) ##### Dependencies - Update dependency com.google.cloud:sdk-platform-java-config to v3.46.3 ([#1801](googleapis/java-logging#1801)) ([d7aa7bc](googleapis/java-logging@d7aa7bc)) - Update dependency com.google.cloud:sdk-platform-java-config to v3.47.0 ([#1803](googleapis/java-logging#1803)) ([5967ffe](googleapis/java-logging@5967ffe)) - Update googleapis/sdk-platform-java action to v2.57.0 ([#1804](googleapis/java-logging#1804)) ([e9a27ec](googleapis/java-logging@e9a27ec)) </details> <details> <summary>googleapis/java-datastore (com.google.cloud:google-cloud-datastore)</summary> ### [`v2.28.1`](https://github.com/googleapis/java-datastore/blob/HEAD/CHANGELOG.md#2281-2025-05-06) ##### Dependencies - Update dependency com.google.cloud:sdk-platform-java-config to v3.47.0 ([#1841](googleapis/java-datastore#1841)) ([ac393e6](googleapis/java-datastore@ac393e6)) - Update googleapis/sdk-platform-java action to v2.57.0 ([#1842](googleapis/java-datastore#1842)) ([0745906](googleapis/java-datastore@0745906)) </details> <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.49.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.49.0): 1.49.0 ### Components #### Configuration at Runtime - ✨ Add process tags as list to remote config payload ([#8705](DataDog/dd-trace-java#8705) - [@amarziali](https://github.com/amarziali)) #### Continuous Integration Visibility - 🐛 Add span propagation for Pekko scheduled tasks ([#8765](DataDog/dd-trace-java#8765) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Update test.retry_reason to use full name of the feature ([#8689](DataDog/dd-trace-java#8689) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - 🧹 Remove unused TestEventsHandler methods ([#8674](DataDog/dd-trace-java#8674) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) #### Dynamic Instrumentation - 🐛 Fix exclude identifiers normalization ([#8742](DataDog/dd-trace-java#8742) - [@jpbempel](https://github.com/jpbempel)) - ✨ Make source file tracking asynchronous ([#8684](DataDog/dd-trace-java#8684) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add scope filtering for symbol extraction ([#8676](DataDog/dd-trace-java#8676) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add support for [@key](https://github.com/key) and [@value](https://github.com/value) for Map filtering ([#8669](DataDog/dd-trace-java#8669) - [@jpbempel](https://github.com/jpbempel)) #### Library Injection - ✨ Add system property to force injection of the tracing library even though multiple javaagents have been detected ([#8697](DataDog/dd-trace-java#8697) - [@cecile75](https://github.com/cecile75)) #### Metrics - ✨ Allow dogstatsd port to be configurable with DD_DOGSTATSD_PORT ([#8693](DataDog/dd-trace-java#8693) - [@randomanderson](https://github.com/randomanderson)) #### Profiling - ✨ Bump ddprof-java to 1.25.1 ([#8750](DataDog/dd-trace-java#8750) - [@jbachorik](https://github.com/jbachorik)) - 🐛 Remove cleanup-on-shutdown for temporary files ([#8746](DataDog/dd-trace-java#8746) - [@jbachorik](https://github.com/jbachorik)) - ✨⚡ Replace a regex-based SMAP parser with a hand-crafted one ([#8730](DataDog/dd-trace-java#8730) - [@jbachorik](https://github.com/jbachorik)) - ✨ Improve error reporting on profiler startup ([#8714](DataDog/dd-trace-java#8714) - [@jbachorik](https://github.com/jbachorik)) - ✨ Exclude ProxyLeakTask exception from exception profiling ([#8666](DataDog/dd-trace-java#8666) - [@jbachorik](https://github.com/jbachorik)) - ✨ Use jvmstat for JDKs 9+ programmatically ([#8641](DataDog/dd-trace-java#8641) - [@MattAlp](https://github.com/MattAlp)) #### Telemetry - ✨ Allow dogstatsd port to be configurable with DD_DOGSTATSD_PORT ([#8693](DataDog/dd-trace-java#8693) - [@randomanderson](https://github.com/randomanderson)) - 🐛 Fix appsec.waf.requests telemetry metric ([#8644](DataDog/dd-trace-java#8644) - [@jandro996](https://github.com/jandro996)) #### Tracer core - ✨ Exclude jackson afterburner dynamic classes from instrumentation ([#8747](DataDog/dd-trace-java#8747) - [@amarziali](https://github.com/amarziali)) - ✨ Introduce Java 8 bytecode bridge for instrumentation API ([#8736](DataDog/dd-trace-java#8736) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ⚡🧹 Use byte-buddy classes optimized for Java8+ ([#8735](DataDog/dd-trace-java#8735) - [@mcculls](https://github.com/mcculls)) - 🐛 Do not set the hibernate or datanucleus span service name when disabled ([#8727](DataDog/dd-trace-java#8727) - [@ygree](https://github.com/ygree)) - ✨ Update bytebuddy and ASM to support JDK 24 ([#8720](DataDog/dd-trace-java#8720) - [@sarahchen6](https://github.com/sarahchen6)) - 🐛 Turn off JDK socket support by default ([#8715](DataDog/dd-trace-java#8715) - [@mcculls](https://github.com/mcculls)) - 🐛 Log warning when trace buffer overflow occurs ([#8712](DataDog/dd-trace-java#8712) - [@ygree](https://github.com/ygree)) - ✨🧪 Introducing an internal integration name ([#8708](DataDog/dd-trace-java#8708) - [@amarziali](https://github.com/amarziali)) - ✨ Add process tags to client stats payload ([#8704](DataDog/dd-trace-java#8704) - [@amarziali](https://github.com/amarziali)) - ✨ Collect process tags for tracing ([#8698](DataDog/dd-trace-java#8698) - [@amarziali](https://github.com/amarziali)) - ✨ Stable Config file: target system properties in process_arguments and support template variables in YamlParser ([#8690](DataDog/dd-trace-java#8690) - [@mtoffl01](https://github.com/mtoffl01)) - ✨⚡ Use prefix trie for proxy ignores ([#8678](DataDog/dd-trace-java#8678) - [@amarziali](https://github.com/amarziali)) - ✨ Allow agent to be automatically injected when running aside Log4J patch agent ([#8648](DataDog/dd-trace-java#8648) - [@paullegranddc](https://github.com/paullegranddc)) - ✨ Use jvmstat for JDKs 9+ programmatically ([#8641](DataDog/dd-trace-java#8641) - [@MattAlp](https://github.com/MattAlp)) #### Tracer internal logging - 🐛 Delete print line ([#8686](DataDog/dd-trace-java#8686) - [@sarahchen6](https://github.com/sarahchen6)) ### Instrumentations #### Akka instrumentation - 🐛 Handle reentrant scope cleanup in Akka/Pekko actor instrumentations ([#8722](DataDog/dd-trace-java#8722) - [@mcculls](https://github.com/mcculls)) #### Apache Spark instrumentation - ✨ Use OpenLineage root parent information to generate trace id ([#8726](DataDog/dd-trace-java#8726) - [@mobuchowski](https://github.com/mobuchowski)) - ✨ Spark job cancellation no longer marks application as failed ([#8701](DataDog/dd-trace-java#8701) - [@paul-laffon-dd](https://github.com/paul-laffon-dd)) #### JDBC instrumentation - 💡 Add support for sybase tds jdbc driver ([#8764](DataDog/dd-trace-java#8764) - [@amarziali](https://github.com/amarziali)) #### Kotlin instrumentation - 🐛 Take defensive copy of parent scope stack when closing nested coroutines ([#8749](DataDog/dd-trace-java#8749) - [@mcculls](https://github.com/mcculls)) #### Reactor instrumentation - ✨⚡ Do not inspect reactor context when not needed ([#8745](DataDog/dd-trace-java#8745) - [@amarziali](https://github.com/amarziali)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 795f347ae34d056efc1194c2f606cee7bca1beea

Add span propagation for Pekko scheduled tasks

ce5a4b2

nikita-tkachenko-datadog added comp: ci visibility Continuous Integration Visibility type: bug Bug report and fix labels May 5, 2025

nikita-tkachenko-datadog marked this pull request as ready for review May 5, 2025 11:02

nikita-tkachenko-datadog requested a review from a team as a code owner May 5, 2025 11:02

amarziali approved these changes May 5, 2025

View reviewed changes

Add instrumentation test and a config knob for Pekko scheduler

3f85a74

nikita-tkachenko-datadog requested a review from a team as a code owner May 5, 2025 14:09

nikita-tkachenko-datadog requested a review from ygree May 5, 2025 14:09

nikita-tkachenko-datadog enabled auto-merge (squash) May 5, 2025 14:09

amarziali approved these changes May 5, 2025

View reviewed changes

Merge branch 'master' into nikita-tkachenko/pekko-scheduler-instrumen…

2ce5ed4

…tation

nikita-tkachenko-datadog merged commit 07d1350 into master May 6, 2025
454 checks passed

nikita-tkachenko-datadog deleted the nikita-tkachenko/pekko-scheduler-instrumentation branch May 6, 2025 10:19

github-actions bot added this to the 1.49.0 milestone May 6, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add span propagation for Pekko scheduled tasks #8765

Add span propagation for Pekko scheduled tasks #8765

Uh oh!

nikita-tkachenko-datadog commented May 5, 2025 •

edited by atlassian bot

Loading

pr-commenter bot commented May 5, 2025 •

edited

Loading

amarziali left a comment

nikita-tkachenko-datadog commented May 5, 2025

Uh oh!

Labels

3 participants

Add span propagation for Pekko scheduled tasks #8765

Add span propagation for Pekko scheduled tasks #8765

Uh oh!

Conversation

nikita-tkachenko-datadog commented May 5, 2025 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

pr-commenter bot commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

amarziali left a comment

Choose a reason for hiding this comment

nikita-tkachenko-datadog commented May 5, 2025

Uh oh!

Labels

3 participants

nikita-tkachenko-datadog commented May 5, 2025 •

edited by atlassian bot

Loading

pr-commenter bot commented May 5, 2025 •

edited

Loading