Perf release gate #9068

bric3 · 2025-06-30T16:59:23Z

What Does This Do

Check then notify if the release do not meet expected thresholds.

Motivation

Ensure performance thresholds.

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

pr-commenter · 2025-06-30T17:33:00Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/r-gate
git_commit_date	1752761628	1752763864
git_commit_sha	`d1794c1`	`df915f0`
release_version	1.52.0-SNAPSHOT~d1794c17af	1.52.0-SNAPSHOT~df915f0a03

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1752765625	1752765625
ci_job_id	1034135474	1034135474
ci_pipeline_id	70870999	70870999
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-1-j5jmyhp9 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-1-j5jmyhp9 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 49 metrics, 10 unstable metrics.

Startup time reports for petclinic

gantt title petclinic - global startup overhead: candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af dateFormat X axisFormat %s section tracing Agent [baseline] (1.043 s) : 0, 1043390 Total [baseline] (10.692 s) : 0, 10692369 Agent [candidate] (1.04 s) : 0, 1039928 Total [candidate] (10.702 s) : 0, 10702188 section appsec Agent [baseline] (1.214 s) : 0, 1213686 Total [baseline] (10.824 s) : 0, 10823993 Agent [candidate] (1.216 s) : 0, 1216395 Total [candidate] (10.77 s) : 0, 10770023 section iast Agent [baseline] (1.186 s) : 0, 1185650 Total [baseline] (10.897 s) : 0, 10896987 Agent [candidate] (1.171 s) : 0, 1171480 Total [candidate] (10.982 s) : 0, 10981897 section profiling Agent [baseline] (1.184 s) : 0, 1183973 Total [baseline] (10.93 s) : 0, 10929792 Agent [candidate] (1.184 s) : 0, 1183632 Total [candidate] (10.937 s) : 0, 10936649

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.043 s	-
Agent	appsec	1.214 s	170.296 ms (16.3%)
Agent	iast	1.186 s	142.26 ms (13.6%)
Agent	profiling	1.184 s	140.582 ms (13.5%)
Total	tracing	10.692 s	-
Total	appsec	10.824 s	131.623 ms (1.2%)
Total	iast	10.897 s	204.618 ms (1.9%)
Total	profiling	10.93 s	237.423 ms (2.2%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.04 s	-
Agent	appsec	1.216 s	176.467 ms (17.0%)
Agent	iast	1.171 s	131.551 ms (12.7%)
Agent	profiling	1.184 s	143.703 ms (13.8%)
Total	tracing	10.702 s	-
Total	appsec	10.77 s	67.836 ms (0.6%)
Total	iast	10.982 s	279.709 ms (2.6%)
Total	profiling	10.937 s	234.461 ms (2.2%)

gantt title petclinic - break down per module: candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af dateFormat X axisFormat %s section tracing crashtracking [baseline] (1.438 ms) : 0, 1438 crashtracking [candidate] (1.428 ms) : 0, 1428 BytebuddyAgent [baseline] (733.131 ms) : 0, 733131 BytebuddyAgent [candidate] (730.932 ms) : 0, 730932 GlobalTracer [baseline] (242.267 ms) : 0, 242267 GlobalTracer [candidate] (241.265 ms) : 0, 241265 AppSec [baseline] (30.577 ms) : 0, 30577 AppSec [candidate] (30.506 ms) : 0, 30506 Debugger [baseline] (6.031 ms) : 0, 6031 Debugger [candidate] (6.016 ms) : 0, 6016 Remote Config [baseline] (662.859 µs) : 0, 663 Remote Config [candidate] (649.928 µs) : 0, 650 Telemetry [baseline] (8.247 ms) : 0, 8247 Telemetry [candidate] (8.192 ms) : 0, 8192 section appsec crashtracking [baseline] (1.439 ms) : 0, 1439 crashtracking [candidate] (1.434 ms) : 0, 1434 BytebuddyAgent [baseline] (748.779 ms) : 0, 748779 BytebuddyAgent [candidate] (750.853 ms) : 0, 750853 GlobalTracer [baseline] (233.218 ms) : 0, 233218 GlobalTracer [candidate] (234.472 ms) : 0, 234472 AppSec [baseline] (166.829 ms) : 0, 166829 AppSec [candidate] (168.234 ms) : 0, 168234 Debugger [baseline] (9.45 ms) : 0, 9450 Debugger [candidate] (7.164 ms) : 0, 7164 Remote Config [baseline] (612.387 µs) : 0, 612 Remote Config [candidate] (607.646 µs) : 0, 608 Telemetry [baseline] (8.895 ms) : 0, 8895 Telemetry [candidate] (8.794 ms) : 0, 8794 IAST [baseline] (23.3 ms) : 0, 23300 IAST [candidate] (23.7 ms) : 0, 23700 section iast crashtracking [baseline] (1.452 ms) : 0, 1452 crashtracking [candidate] (1.427 ms) : 0, 1427 BytebuddyAgent [baseline] (856.305 ms) : 0, 856305 BytebuddyAgent [candidate] (845.804 ms) : 0, 845804 GlobalTracer [baseline] (234.801 ms) : 0, 234801 GlobalTracer [candidate] (230.774 ms) : 0, 230774 AppSec [baseline] (28.988 ms) : 0, 28988 AppSec [candidate] (27.562 ms) : 0, 27562 Debugger [baseline] (6.633 ms) : 0, 6633 Debugger [candidate] (6.737 ms) : 0, 6737 Remote Config [baseline] (594.273 µs) : 0, 594 Remote Config [candidate] (597.083 µs) : 0, 597 Telemetry [baseline] (8.025 ms) : 0, 8025 Telemetry [candidate] (7.921 ms) : 0, 7921 IAST [baseline] (27.734 ms) : 0, 27734 IAST [candidate] (29.827 ms) : 0, 29827 section profiling crashtracking [baseline] (1.428 ms) : 0, 1428 crashtracking [candidate] (1.409 ms) : 0, 1409 BytebuddyAgent [baseline] (759.06 ms) : 0, 759060 BytebuddyAgent [candidate] (759.498 ms) : 0, 759498 GlobalTracer [baseline] (220.833 ms) : 0, 220833 GlobalTracer [candidate] (220.32 ms) : 0, 220320 AppSec [baseline] (30.4 ms) : 0, 30400 AppSec [candidate] (30.339 ms) : 0, 30339 Debugger [baseline] (6.293 ms) : 0, 6293 Debugger [candidate] (6.322 ms) : 0, 6322 Remote Config [baseline] (700.313 µs) : 0, 700 Remote Config [candidate] (675.378 µs) : 0, 675 Telemetry [baseline] (10.974 ms) : 0, 10974 Telemetry [candidate] (12.394 ms) : 0, 12394 ProfilingAgent [baseline] (106.018 ms) : 0, 106018 ProfilingAgent [candidate] (104.396 ms) : 0, 104396 Profiling [baseline] (106.04 ms) : 0, 106040 Profiling [candidate] (104.418 ms) : 0, 104418

Startup time reports for insecure-bank

gantt title insecure-bank - global startup overhead: candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af dateFormat X axisFormat %s section tracing Agent [baseline] (1.042 s) : 0, 1041547 Total [baseline] (8.636 s) : 0, 8635628 Agent [candidate] (1.037 s) : 0, 1037324 Total [candidate] (8.602 s) : 0, 8601976 section iast Agent [baseline] (1.171 s) : 0, 1171232 Total [baseline] (9.282 s) : 0, 9282471 Agent [candidate] (1.19 s) : 0, 1190343 Total [candidate] (9.35 s) : 0, 9349969

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.042 s	-
Agent	iast	1.171 s	129.685 ms (12.5%)
Total	tracing	8.636 s	-
Total	iast	9.282 s	646.843 ms (7.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.037 s	-
Agent	iast	1.19 s	153.02 ms (14.8%)
Total	tracing	8.602 s	-
Total	iast	9.35 s	747.993 ms (8.7%)

gantt title insecure-bank - break down per module: candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af dateFormat X axisFormat %s section tracing crashtracking [baseline] (1.43 ms) : 0, 1430 crashtracking [candidate] (1.432 ms) : 0, 1432 BytebuddyAgent [baseline] (731.047 ms) : 0, 731047 BytebuddyAgent [candidate] (728.97 ms) : 0, 728970 GlobalTracer [baseline] (242.523 ms) : 0, 242523 GlobalTracer [candidate] (240.887 ms) : 0, 240887 AppSec [baseline] (30.641 ms) : 0, 30641 AppSec [candidate] (30.305 ms) : 0, 30305 Debugger [baseline] (6.037 ms) : 0, 6037 Debugger [candidate] (5.989 ms) : 0, 5989 Remote Config [baseline] (657.752 µs) : 0, 658 Remote Config [candidate] (654.164 µs) : 0, 654 Telemetry [baseline] (8.3 ms) : 0, 8300 Telemetry [candidate] (8.183 ms) : 0, 8183 section iast crashtracking [baseline] (1.425 ms) : 0, 1425 crashtracking [candidate] (1.453 ms) : 0, 1453 BytebuddyAgent [baseline] (845.701 ms) : 0, 845701 BytebuddyAgent [candidate] (861.146 ms) : 0, 861146 GlobalTracer [baseline] (230.673 ms) : 0, 230673 GlobalTracer [candidate] (233.319 ms) : 0, 233319 AppSec [baseline] (28.144 ms) : 0, 28144 AppSec [candidate] (27.675 ms) : 0, 27675 Debugger [baseline] (8.328 ms) : 0, 8328 Debugger [candidate] (6.75 ms) : 0, 6750 Remote Config [baseline] (589.455 µs) : 0, 589 Remote Config [candidate] (615.116 µs) : 0, 615 Telemetry [baseline] (7.962 ms) : 0, 7962 Telemetry [candidate] (8.071 ms) : 0, 8071 IAST [baseline] (27.497 ms) : 0, 27497 IAST [candidate] (30.198 ms) : 0, 30198

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/r-gate
git_commit_date	1752761628	1752763864
git_commit_sha	`d1794c1`	`df915f0`
release_version	1.52.0-SNAPSHOT~d1794c17af	1.52.0-SNAPSHOT~df915f0a03

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1752765370	1752765370
ci_job_id	1034135475	1034135475
ci_pipeline_id	70870999	70870999
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-0-hrg2yowu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-0-hrg2yowu 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 2 performance regressions! Performance is the same for 9 metrics, 12 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:insecure-bank:tracing:high_load	worse [+334.443µs; +587.842µs] or [+4.381%; +7.700%]	unstable [-109.476op/s; +41.038op/s] or [-18.043%; +6.764%]	8.095ms	572.531op/s	7.634ms	606.750op/s
scenario:load:insecure-bank:iast:high_load	worse [+304.279µs; +637.376µs] or [+3.341%; +6.998%]	unstable [-83.520op/s; +33.833op/s] or [-16.392%; +6.640%]	9.579ms	484.688op/s	9.109ms	509.531op/s
scenario:load:petclinic:appsec:high_load	better [-4.214ms; -3.277ms] or [-8.415%; -6.545%]	unstable [+0.324op/s; +14.776op/s] or [+0.347%; +15.801%]	46.325ms	101.062op/s	50.070ms	93.513op/s

Request duration reports for insecure-bank

gantt title insecure-bank - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af dateFormat X axisFormat %s section baseline no_agent (4.452 ms) : 4391, 4512 . : milestone, 4452, iast (9.109 ms) : 8960, 9257 . : milestone, 9109, iast_FULL (14.038 ms) : 13761, 14315 . : milestone, 14038, iast_GLOBAL (10.033 ms) : 9840, 10227 . : milestone, 10033, profiling (8.733 ms) : 8583, 8884 . : milestone, 8733, tracing (7.634 ms) : 7518, 7750 . : milestone, 7634, section candidate no_agent (4.462 ms) : 4412, 4512 . : milestone, 4462, iast (9.579 ms) : 9419, 9740 . : milestone, 9579, iast_FULL (13.78 ms) : 13505, 14055 . : milestone, 13780, iast_GLOBAL (10.197 ms) : 10015, 10379 . : milestone, 10197, profiling (9.002 ms) : 8862, 9143 . : milestone, 9002, tracing (8.095 ms) : 7976, 8215 . : milestone, 8095,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.452 ms [4.391 ms, 4.512 ms]	-
iast	9.109 ms [8.96 ms, 9.257 ms]	4.657 ms (104.6%)
iast_FULL	14.038 ms [13.761 ms, 14.315 ms]	9.587 ms (215.4%)
iast_GLOBAL	10.033 ms [9.84 ms, 10.227 ms]	5.582 ms (125.4%)
profiling	8.733 ms [8.583 ms, 8.884 ms]	4.282 ms (96.2%)
tracing	7.634 ms [7.518 ms, 7.75 ms]	3.183 ms (71.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	4.462 ms [4.412 ms, 4.512 ms]	-
iast	9.579 ms [9.419 ms, 9.74 ms]	5.118 ms (114.7%)
iast_FULL	13.78 ms [13.505 ms, 14.055 ms]	9.318 ms (208.8%)
iast_GLOBAL	10.197 ms [10.015 ms, 10.379 ms]	5.735 ms (128.5%)
profiling	9.002 ms [8.862 ms, 9.143 ms]	4.541 ms (101.8%)
tracing	8.095 ms [7.976 ms, 8.215 ms]	3.633 ms (81.4%)

Request duration reports for petclinic

gantt title petclinic - request duration [CI 0.99] : candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af dateFormat X axisFormat %s section baseline no_agent (37.981 ms) : 37667, 38296 . : milestone, 37981, appsec (50.07 ms) : 49607, 50534 . : milestone, 50070, code_origins (44.599 ms) : 44209, 44989 . : milestone, 44599, iast (44.459 ms) : 44079, 44839 . : milestone, 44459, profiling (47.137 ms) : 46689, 47586 . : milestone, 47137, tracing (44.719 ms) : 44338, 45100 . : milestone, 44719, section candidate no_agent (37.54 ms) : 37237, 37844 . : milestone, 37540, appsec (46.325 ms) : 45920, 46730 . : milestone, 46325, code_origins (44.838 ms) : 44464, 45213 . : milestone, 44838, iast (44.708 ms) : 44315, 45101 . : milestone, 44708, profiling (47.536 ms) : 47083, 47990 . : milestone, 47536, tracing (44.137 ms) : 43765, 44508 . : milestone, 44137,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.981 ms [37.667 ms, 38.296 ms]	-
appsec	50.07 ms [49.607 ms, 50.534 ms]	12.089 ms (31.8%)
code_origins	44.599 ms [44.209 ms, 44.989 ms]	6.618 ms (17.4%)
iast	44.459 ms [44.079 ms, 44.839 ms]	6.477 ms (17.1%)
profiling	47.137 ms [46.689 ms, 47.586 ms]	9.156 ms (24.1%)
tracing	44.719 ms [44.338 ms, 45.1 ms]	6.738 ms (17.7%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	37.54 ms [37.237 ms, 37.844 ms]	-
appsec	46.325 ms [45.92 ms, 46.73 ms]	8.785 ms (23.4%)
code_origins	44.838 ms [44.464 ms, 45.213 ms]	7.298 ms (19.4%)
iast	44.708 ms [44.315 ms, 45.101 ms]	7.168 ms (19.1%)
profiling	47.536 ms [47.083 ms, 47.99 ms]	9.996 ms (26.6%)
tracing	44.137 ms [43.765 ms, 44.508 ms]	6.597 ms (17.6%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	bdu/r-gate
git_commit_date	1752761628	1752763864
git_commit_sha	`d1794c1`	`df915f0`
release_version	1.52.0-SNAPSHOT~d1794c17af	1.52.0-SNAPSHOT~df915f0a03

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1752765790	1752765790
ci_job_id	1034135476	1034135476
ci_pipeline_id	70870999	70870999
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-zfyrx7zua-project-304-concurrent-2-c35tsyqg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-zfyrx7zua-project-304-concurrent-2-c35tsyqg 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for tomcat

gantt title tomcat - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af dateFormat X axisFormat %s section baseline no_agent (1.476 ms) : 1464, 1487 . : milestone, 1476, appsec (3.652 ms) : 3434, 3871 . : milestone, 3652, iast (2.202 ms) : 2139, 2264 . : milestone, 2202, iast_GLOBAL (2.236 ms) : 2174, 2299 . : milestone, 2236, profiling (2.035 ms) : 1985, 2085 . : milestone, 2035, tracing (2.012 ms) : 1963, 2061 . : milestone, 2012, section candidate no_agent (1.476 ms) : 1465, 1488 . : milestone, 1476, appsec (3.573 ms) : 3361, 3786 . : milestone, 3573, iast (2.209 ms) : 2146, 2272 . : milestone, 2209, iast_GLOBAL (2.244 ms) : 2181, 2308 . : milestone, 2244, profiling (2.044 ms) : 1994, 2095 . : milestone, 2044, tracing (2.02 ms) : 1971, 2068 . : milestone, 2020,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.464 ms, 1.487 ms]	-
appsec	3.652 ms [3.434 ms, 3.871 ms]	2.177 ms (147.5%)
iast	2.202 ms [2.139 ms, 2.264 ms]	725.646 µs (49.2%)
iast_GLOBAL	2.236 ms [2.174 ms, 2.299 ms]	760.395 µs (51.5%)
profiling	2.035 ms [1.985 ms, 2.085 ms]	558.867 µs (37.9%)
tracing	2.012 ms [1.963 ms, 2.061 ms]	536.156 µs (36.3%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.476 ms [1.465 ms, 1.488 ms]	-
appsec	3.573 ms [3.361 ms, 3.786 ms]	2.097 ms (142.1%)
iast	2.209 ms [2.146 ms, 2.272 ms]	732.582 µs (49.6%)
iast_GLOBAL	2.244 ms [2.181 ms, 2.308 ms]	768.271 µs (52.0%)
profiling	2.044 ms [1.994 ms, 2.095 ms]	568.107 µs (38.5%)
tracing	2.02 ms [1.971 ms, 2.068 ms]	543.416 µs (36.8%)

Execution time for biojava

gantt title biojava - execution time [CI 0.99] : candidate=1.52.0-SNAPSHOT~df915f0a03, baseline=1.52.0-SNAPSHOT~d1794c17af dateFormat X axisFormat %s section baseline no_agent (14.758 s) : 14758000, 14758000 . : milestone, 14758000, appsec (14.983 s) : 14983000, 14983000 . : milestone, 14983000, iast (18.089 s) : 18089000, 18089000 . : milestone, 18089000, iast_GLOBAL (18.002 s) : 18002000, 18002000 . : milestone, 18002000, profiling (16.055 s) : 16055000, 16055000 . : milestone, 16055000, tracing (14.959 s) : 14959000, 14959000 . : milestone, 14959000, section candidate no_agent (15.672 s) : 15672000, 15672000 . : milestone, 15672000, appsec (14.891 s) : 14891000, 14891000 . : milestone, 14891000, iast (18.632 s) : 18632000, 18632000 . : milestone, 18632000, iast_GLOBAL (18.276 s) : 18276000, 18276000 . : milestone, 18276000, profiling (15.21 s) : 15210000, 15210000 . : milestone, 15210000, tracing (15.015 s) : 15015000, 15015000 . : milestone, 15015000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.758 s [14.758 s, 14.758 s]	-
appsec	14.983 s [14.983 s, 14.983 s]	225.0 ms (1.5%)
iast	18.089 s [18.089 s, 18.089 s]	3.331 s (22.6%)
iast_GLOBAL	18.002 s [18.002 s, 18.002 s]	3.244 s (22.0%)
profiling	16.055 s [16.055 s, 16.055 s]	1.297 s (8.8%)
tracing	14.959 s [14.959 s, 14.959 s]	201.0 ms (1.4%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.672 s [15.672 s, 15.672 s]	-
appsec	14.891 s [14.891 s, 14.891 s]	-781.0 ms (-5.0%)
iast	18.632 s [18.632 s, 18.632 s]	2.96 s (18.9%)
iast_GLOBAL	18.276 s [18.276 s, 18.276 s]	2.604 s (16.6%)
profiling	15.21 s [15.21 s, 15.21 s]	-462.0 ms (-2.9%)
tracing	15.015 s [15.015 s, 15.015 s]	-657.0 ms (-4.2%)

ddyurchenko

Looks good from my side! 🎉
Need also @igoragoli review for the final approval.

igoragoli

Thanks @bric3! 🙌

There are just some points regarding startup:petclinic.* thresholds that need to be addressed before merging.

.gitlab/benchmarks/bp-runner.fail-on-breach.yml

.gitlab/macrobenchmarks.yml

…can debug

igoragoli

Looks good, thanks Brice!

I think it's a good idea to include the source for the SLOs in the thresholds file, nice.

bric3 · 2025-07-11T14:12:06Z

.gitlab/macrobenchmarks.yml

- when: always
- - when: manual
- allow_failure: true
+ - when: on_success # TODO: PLEASE revert before merging the PR


todo:

To revert before merging

ddyurchenko · 2025-07-14T15:30:27Z

Still see some yellow in results, so while job won't block releases, it will send warning messages in Java guild channel.
Augusto practically finalized the changes to reporting (so now confidence intervals and thresholds are clearly displayed, incl. warning threshold). I propose to rebuild Java image, so the changes are included, and update the SLOs once more for p50 latency, p99 latency and startup time mean value (execution_time metric), so they are no longer in yellow zone.

SLO breach check | SLO breach check | #### high_load--only-tracing SLO breach check | SLO breach check | - 🟩 `throughput` 1[250](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1025486085#L250).55 op/s > 1100.00 op/s SLO breach check | SLO breach check | #### high_load--otel-latest SLO breach check | SLO breach check | - 🟩 `throughput` 1245.11 op/s > 1100.00 op/s SLO breach check | SLO breach check | #### normal_operation--only-tracing SLO breach check | SLO breach check | - 🟩 `agg_http_req_duration_p50` 2.12 ms < 2.36 ms SLO breach check | - 🟨 `agg_http_req_duration_p99` 7.10 ms < 7.89 ms SLO breach check | SLO breach check | #### normal_operation--otel-latest SLO breach check | SLO breach check | - 🟨 `agg_http_req_duration_p50` 2.12 ms < 2.34 ms SLO breach check | - 🟨 `agg_http_req_duration_p99` 8.75 ms < 9.50 ms SLO breach check | SLO breach check | #### startup:petclinic:appsec:GlobalTracer SLO breach check | SLO breach check | - 🟨 `execution_time` 235.01 ms < [260](https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1025486085#L260).00 ms SLO breach check | SLO breach check | #### startup:petclinic:iast:GlobalTracer SLO breach check | SLO breach check | - 🟩 `execution_time` 231.70 ms < 260.00 ms SLO breach check | SLO breach check | #### startup:petclinic:profiling:GlobalTracer SLO breach check | SLO breach check | - 🟨 `execution_time` 361.27 ms < 368.00 ms SLO breach check | SLO breach check | #### startup:petclinic:tracing:GlobalTracer SLO breach check | SLO breach check | - 🟨 `execution_time` 243.13 ms < 260.00 ms SLO breach check | SLO breach check | --- SLO breach check | SLO breach check | Legend: SLO breach check | - 🟩 pass SLO breach check | - 🟥 breach SLO breach check | - 🟨 warning SLO breach check | - (unstable) unstable

igoragoli · 2025-07-16T12:16:09Z

Augusto practically finalized the changes to reporting

Changes for reporting done! 🙂

I'm updating the registry.ddbuild.io/images/benchmarking-platform-tools-ubuntu:latest image (the one used in the check-slo-breaches job across the board) here: https://gitlab.ddbuild.io/DataDog/benchmarking-platform-tools/-/jobs/1031270236

ddyurchenko · 2025-07-16T13:35:43Z

Thanks @igoragoli ! I reran the job https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1031460223, based on its results, will provide suggestions to @bric3 .

ddyurchenko · 2025-07-16T13:39:14Z

.gitlab/benchmarks/bp-runner.fail-on-breach.yml

+ # https://datadoghq.atlassian.net/wiki/x/LgI1LgE#Specification
+ # Measurements
+ # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario
+ scenarios:


Based on the results of https://gitlab.ddbuild.io/DataDog/apm-reliability/dd-trace-java/-/jobs/1031460223, I suggest to update the SLOs to the following values:

# Standard macrobenchmarks # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=normal_operation%2Fonly-tracing&trendsType=scenario - name: normal_operation/only-tracing thresholds: - agg_http_req_duration_p50 < 2.36 ms - agg_http_req_duration_p99 < 7.89 ms # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=normal_operation%2Fotel-latest&trendsType=scenario - name: normal_operation/otel-latest thresholds: - agg_http_req_duration_p50 < 2.5 ms - agg_http_req_duration_p99 < 10 ms # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=high_load%2Fonly-tracing&trendsType=scenario - name: high_load/only-tracing thresholds: - throughput > 1100.0 op/s # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=high_load%2Fotel-latest&trendsType=scenario - name: high_load/otel-latest thresholds: - throughput > 1100.0 op/s # Startup macrobenchmarks # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Atracing%3AGlobalTracer&trendsType=scenario # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aappsec%3AGlobalTracer&trendsType=scenario # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aiast%3AGlobalTracer&trendsType=scenario - name: "startup:petclinic:(tracing|appsec|iast):GlobalTracer" thresholds: - execution_time < 280 ms # https://benchmarking.us1.prod.dog/trends?projectId=4&branch=master&trendsTab=per_scenario&scenario=startup%3Apetclinic%3Aprofiling%3AGlobalTracer&trendsType=scenario - name: "startup:petclinic:profiling:GlobalTracer" thresholds: - execution_time < 420 ms

Another tweak

ddyurchenko

Need to update some of SLO thresholds before merge.

See https://github.com/DataDog/dd-trace-java/pull/9068/files#r2210474360

…robenchmarks)

bric3 requested a review from a team as a code owner June 30, 2025 16:59

bric3 requested review from colin-higgins and removed request for a team June 30, 2025 16:59

bric3 added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling labels Jun 30, 2025

bric3 marked this pull request as draft June 30, 2025 16:59

bric3 changed the title ~~chore(ci): Basic slo breach prototype~~ Perf release gate Jun 30, 2025

ddyurchenko approved these changes Jul 1, 2025

View reviewed changes

ddyurchenko requested a review from igoragoli July 1, 2025 13:43

bric3 requested a review from a team July 1, 2025 14:07

igoragoli requested changes Jul 1, 2025

View reviewed changes

.gitlab/benchmarks/bp-runner.fail-on-breach.yml Outdated Show resolved Hide resolved

.gitlab/benchmarks/bp-runner.fail-on-breach.yml Outdated Show resolved Hide resolved

.gitlab/macrobenchmarks.yml Show resolved Hide resolved

bric3 requested a review from a team July 2, 2025 15:24

bric3 force-pushed the bdu/r-gate branch 7 times, most recently from 7e61d82 to feb60ef Compare July 10, 2025 12:48

bric3 added 3 commits July 11, 2025 11:35

chore(ci): Basic slo breach prototype

fdd30e5

chore(ci): PR review

6407c42

chore: Collect reports from benchmarks

736aefd

bric3 force-pushed the bdu/r-gate branch from feb60ef to 736aefd Compare July 11, 2025 09:35

ddyurchenko added 2 commits July 11, 2025 15:28

tweak: Include benchmarkType in file name

361aaf6

tweak: Store artifacts that were used for checking regression, so we …

92f9783

…can debug

ddyurchenko self-requested a review July 11, 2025 14:39

chore(ci): Tweak thresholds

67435a9

bric3 marked this pull request as ready for review July 11, 2025 15:45

igoragoli approved these changes Jul 11, 2025

View reviewed changes

bric3 commented Jul 11, 2025

View reviewed changes

ddyurchenko reviewed Jul 16, 2025

View reviewed changes

ddyurchenko requested changes Jul 16, 2025

View reviewed changes

bric3 added 3 commits July 17, 2025 11:26

chore(ci): Tweak to recommended thresholds

616ae97

See https://github.com/DataDog/dd-trace-java/pull/9068/files#r2210474360

Merge remote-tracking branch 'origin/master' into bdu/r-gate

0708dfb

chore(ci): Another tweak to recommended thresholds

98f9a31

See https://github.com/DataDog/dd-trace-java/pull/9068/files#r2210474360

ddyurchenko approved these changes Jul 17, 2025

View reviewed changes

bric3 and others added 2 commits July 17, 2025 14:49

chore(ci): Revert hack to run the release gate (it needed the all mac…

098e012

…robenchmarks)

Merge branch 'master' into bdu/r-gate

df915f0

bric3 enabled auto-merge (squash) July 22, 2025 10:08

bric3 merged commit 4e4c286 into master Jul 22, 2025
503 checks passed

bric3 deleted the bdu/r-gate branch July 22, 2025 10:33

github-actions bot added this to the 1.52.0 milestone Jul 22, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Perf release gate #9068

Perf release gate #9068

Uh oh!

bric3 commented Jun 30, 2025 •

edited

Loading

pr-commenter bot commented Jun 30, 2025 •

edited

Loading

ddyurchenko left a comment

igoragoli left a comment

Uh oh!

Uh oh!

Uh oh!

igoragoli left a comment

bric3 Jul 11, 2025

ddyurchenko commented Jul 14, 2025

igoragoli commented Jul 16, 2025 •

edited

Loading

ddyurchenko commented Jul 16, 2025

ddyurchenko Jul 16, 2025

bric3 Jul 17, 2025

bric3 Jul 17, 2025

ddyurchenko left a comment

Uh oh!

Labels

3 participants

Perf release gate #9068

Perf release gate #9068

Uh oh!

Conversation

bric3 commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

pr-commenter bot commented Jun 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

ddyurchenko left a comment

Choose a reason for hiding this comment

igoragoli left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

igoragoli left a comment

Choose a reason for hiding this comment

bric3 Jul 11, 2025

Choose a reason for hiding this comment

ddyurchenko commented Jul 14, 2025

igoragoli commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ddyurchenko commented Jul 16, 2025

ddyurchenko Jul 16, 2025

Choose a reason for hiding this comment

bric3 Jul 17, 2025

Choose a reason for hiding this comment

bric3 Jul 17, 2025

Choose a reason for hiding this comment

ddyurchenko left a comment

Choose a reason for hiding this comment

Uh oh!

Labels

3 participants

bric3 commented Jun 30, 2025 •

edited

Loading

pr-commenter bot commented Jun 30, 2025 •

edited

Loading

igoragoli commented Jul 16, 2025 •

edited

Loading