Skip to content

Conversation

AlexeyKuznetsov-DD
Copy link
Contributor

What Does This Do

Attempt to check with bigger timeout.

Motivation

Investigation of flaky test.

Additional Notes

DO NOT MERGE

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD self-assigned this Jun 27, 2025
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD added the tag: do not merge Do not merge changes label Jun 27, 2025
@pr-commenter
Copy link

pr-commenter bot commented Jun 27, 2025

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master alexeyk/maven-smoke-test-timeout
git_commit_date 1751046924 1751147372
git_commit_sha 46d0949 d160cb7
release_version 1.51.0-SNAPSHOT~46d09492c9 1.51.0-SNAPSHOT~d160cb7b30
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1751154724 1751154724
ci_job_id 1003861087 1003861087
ci_pipeline_id 69077238 69077238
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-6vsgw5eg-project-304-concurrent-0-2tsgyiyo 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-6vsgw5eg-project-304-concurrent-0-2tsgyiyo 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 7 unstable metrics.

Startup time reports for insecure-bank
gantt title insecure-bank - global startup overhead: candidate=1.51.0-SNAPSHOT~d160cb7b30, baseline=1.51.0-SNAPSHOT~46d09492c9 dateFormat X axisFormat %s section tracing Agent [baseline] (1.002 s) : 0, 1002162 Total [baseline] (8.566 s) : 0, 8566433 Agent [candidate] (996.968 ms) : 0, 996968 Total [candidate] (8.59 s) : 0, 8590181 section iast Agent [baseline] (1.138 s) : 0, 1138474 Total [baseline] (9.254 s) : 0, 9254013 Agent [candidate] (1.148 s) : 0, 1148421 Total [candidate] (9.288 s) : 0, 9287660 
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.002 s -
Agent iast 1.138 s 136.312 ms (13.6%)
Total tracing 8.566 s -
Total iast 9.254 s 687.58 ms (8.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 996.968 ms -
Agent iast 1.148 s 151.454 ms (15.2%)
Total tracing 8.59 s -
Total iast 9.288 s 697.479 ms (8.1%)
gantt title insecure-bank - break down per module: candidate=1.51.0-SNAPSHOT~d160cb7b30, baseline=1.51.0-SNAPSHOT~46d09492c9 dateFormat X axisFormat %s section tracing BytebuddyAgent [baseline] (693.42 ms) : 0, 693420 BytebuddyAgent [candidate] (688.777 ms) : 0, 688777 GlobalTracer [baseline] (242.478 ms) : 0, 242478 GlobalTracer [candidate] (242.142 ms) : 0, 242142 AppSec [baseline] (30.334 ms) : 0, 30334 AppSec [candidate] (30.382 ms) : 0, 30382 Debugger [baseline] (6.072 ms) : 0, 6072 Debugger [candidate] (6.033 ms) : 0, 6033 Remote Config [baseline] (683.238 µs) : 0, 683 Remote Config [candidate] (683.402 µs) : 0, 683 Telemetry [baseline] (8.246 ms) : 0, 8246 Telemetry [candidate] (8.172 ms) : 0, 8172 section iast BytebuddyAgent [baseline] (813.907 ms) : 0, 813907 BytebuddyAgent [candidate] (820.437 ms) : 0, 820437 GlobalTracer [baseline] (233.396 ms) : 0, 233396 GlobalTracer [candidate] (235.752 ms) : 0, 235752 AppSec [baseline] (28.536 ms) : 0, 28536 AppSec [candidate] (28.998 ms) : 0, 28998 Debugger [baseline] (5.845 ms) : 0, 5845 Debugger [candidate] (5.908 ms) : 0, 5908 Remote Config [baseline] (594.275 µs) : 0, 594 Remote Config [candidate] (595.521 µs) : 0, 596 Telemetry [baseline] (8.01 ms) : 0, 8010 Telemetry [candidate] (8.272 ms) : 0, 8272 IAST [baseline] (27.242 ms) : 0, 27242 IAST [candidate] (27.448 ms) : 0, 27448 
Loading
Startup time reports for petclinic
gantt title petclinic - global startup overhead: candidate=1.51.0-SNAPSHOT~d160cb7b30, baseline=1.51.0-SNAPSHOT~46d09492c9 dateFormat X axisFormat %s section tracing Agent [baseline] (999.518 ms) : 0, 999518 Total [baseline] (10.795 s) : 0, 10795384 Agent [candidate] (996.64 ms) : 0, 996640 Total [candidate] (10.663 s) : 0, 10662624 section appsec Agent [baseline] (1.175 s) : 0, 1175075 Total [baseline] (10.741 s) : 0, 10741003 Agent [candidate] (1.181 s) : 0, 1181121 Total [candidate] (10.767 s) : 0, 10767484 section iast Agent [baseline] (1.139 s) : 0, 1138671 Total [baseline] (10.931 s) : 0, 10931065 Agent [candidate] (1.131 s) : 0, 1131138 Total [candidate] (10.808 s) : 0, 10807860 section profiling Agent [baseline] (1.251 s) : 0, 1250911 Total [baseline] (11.034 s) : 0, 11033584 Agent [candidate] (1.245 s) : 0, 1245126 Total [candidate] (10.986 s) : 0, 10986389 
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 999.518 ms -
Agent appsec 1.175 s 175.557 ms (17.6%)
Agent iast 1.139 s 139.154 ms (13.9%)
Agent profiling 1.251 s 251.393 ms (25.2%)
Total tracing 10.795 s -
Total appsec 10.741 s -54.381 ms (-0.5%)
Total iast 10.931 s 135.682 ms (1.3%)
Total profiling 11.034 s 238.2 ms (2.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 996.64 ms -
Agent appsec 1.181 s 184.481 ms (18.5%)
Agent iast 1.131 s 134.498 ms (13.5%)
Agent profiling 1.245 s 248.486 ms (24.9%)
Total tracing 10.663 s -
Total appsec 10.767 s 104.86 ms (1.0%)
Total iast 10.808 s 145.237 ms (1.4%)
Total profiling 10.986 s 323.766 ms (3.0%)
gantt title petclinic - break down per module: candidate=1.51.0-SNAPSHOT~d160cb7b30, baseline=1.51.0-SNAPSHOT~46d09492c9 dateFormat X axisFormat %s section tracing BytebuddyAgent [baseline] (690.245 ms) : 0, 690245 BytebuddyAgent [candidate] (687.948 ms) : 0, 687948 GlobalTracer [baseline] (243.191 ms) : 0, 243191 GlobalTracer [candidate] (242.627 ms) : 0, 242627 AppSec [baseline] (30.322 ms) : 0, 30322 AppSec [candidate] (30.372 ms) : 0, 30372 Debugger [baseline] (6.012 ms) : 0, 6012 Debugger [candidate] (6.026 ms) : 0, 6026 Remote Config [baseline] (669.943 µs) : 0, 670 Remote Config [candidate] (674.556 µs) : 0, 675 Telemetry [baseline] (8.19 ms) : 0, 8190 Telemetry [candidate] (8.183 ms) : 0, 8183 section appsec BytebuddyAgent [baseline] (711.403 ms) : 0, 711403 BytebuddyAgent [candidate] (715.565 ms) : 0, 715565 GlobalTracer [baseline] (235.535 ms) : 0, 235535 GlobalTracer [candidate] (236.539 ms) : 0, 236539 AppSec [baseline] (170.719 ms) : 0, 170719 AppSec [candidate] (171.457 ms) : 0, 171457 Debugger [baseline] (5.775 ms) : 0, 5775 Debugger [candidate] (5.807 ms) : 0, 5807 Remote Config [baseline] (598.071 µs) : 0, 598 Remote Config [candidate] (616.365 µs) : 0, 616 Telemetry [baseline] (8.124 ms) : 0, 8124 Telemetry [candidate] (8.088 ms) : 0, 8088 IAST [baseline] (22.116 ms) : 0, 22116 IAST [candidate] (22.144 ms) : 0, 22144 section iast BytebuddyAgent [baseline] (813.071 ms) : 0, 813071 BytebuddyAgent [candidate] (807.942 ms) : 0, 807942 GlobalTracer [baseline] (234.485 ms) : 0, 234485 GlobalTracer [candidate] (232.521 ms) : 0, 232521 AppSec [baseline] (26.172 ms) : 0, 26172 AppSec [candidate] (27.628 ms) : 0, 27628 Debugger [baseline] (5.88 ms) : 0, 5880 Debugger [candidate] (5.82 ms) : 0, 5820 Remote Config [baseline] (596.49 µs) : 0, 596 Remote Config [candidate] (582.127 µs) : 0, 582 Telemetry [baseline] (8.01 ms) : 0, 8010 Telemetry [candidate] (7.951 ms) : 0, 7951 IAST [baseline] (29.594 ms) : 0, 29594 IAST [candidate] (27.916 ms) : 0, 27916 section profiling BytebuddyAgent [baseline] (682.253 ms) : 0, 682253 BytebuddyAgent [candidate] (679.643 ms) : 0, 679643 GlobalTracer [baseline] (362.274 ms) : 0, 362274 GlobalTracer [candidate] (361.124 ms) : 0, 361124 AppSec [baseline] (31.706 ms) : 0, 31706 AppSec [candidate] (30.769 ms) : 0, 30769 Debugger [baseline] (11.225 ms) : 0, 11225 Debugger [candidate] (12.795 ms) : 0, 12795 Remote Config [baseline] (690.562 µs) : 0, 691 Remote Config [candidate] (1.345 ms) : 0, 1345 Telemetry [baseline] (9.506 ms) : 0, 9506 Telemetry [candidate] (8.098 ms) : 0, 8098 ProfilingAgent [baseline] (104.297 ms) : 0, 104297 ProfilingAgent [candidate] (102.618 ms) : 0, 102618 Profiling [baseline] (104.322 ms) : 0, 104322 Profiling [candidate] (102.643 ms) : 0, 102643 
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master alexeyk/maven-smoke-test-timeout
git_commit_date 1751046924 1751147372
git_commit_sha 46d0949 d160cb7
release_version 1.51.0-SNAPSHOT~46d09492c9 1.51.0-SNAPSHOT~d160cb7b30
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1751154395 1751154395
ci_job_id 1003861088 1003861088
ci_pipeline_id 69077238 69077238
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-oyrxellh-project-304-concurrent-0-wn80urwk 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-oyrxellh-project-304-concurrent-0-wn80urwk 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 12 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:profiling:high_load better
[-2.209ms; -1.288ms] or [-4.517%; -2.634%]
unstable
[-1.900op/s; +11.563op/s] or [-1.986%; +12.085%]
47.157ms 100.506op/s 48.905ms 95.675op/s
Request duration reports for insecure-bank
gantt title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~d160cb7b30, baseline=1.51.0-SNAPSHOT~46d09492c9 dateFormat X axisFormat %s section baseline no_agent (4.29 ms) : 4242, 4339 . : milestone, 4290, iast (9.258 ms) : 9105, 9410 . : milestone, 9258, iast_FULL (14.161 ms) : 13870, 14452 . : milestone, 14161, iast_GLOBAL (10.103 ms) : 9930, 10276 . : milestone, 10103, profiling (8.967 ms) : 8817, 9117 . : milestone, 8967, tracing (8.007 ms) : 7892, 8122 . : milestone, 8007, section candidate no_agent (4.334 ms) : 4284, 4384 . : milestone, 4334, iast (9.128 ms) : 8979, 9277 . : milestone, 9128, iast_FULL (14.106 ms) : 13825, 14387 . : milestone, 14106, iast_GLOBAL (10.32 ms) : 10141, 10499 . : milestone, 10320, profiling (8.997 ms) : 8853, 9141 . : milestone, 8997, tracing (7.736 ms) : 7624, 7849 . : milestone, 7736, 
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.29 ms [4.242 ms, 4.339 ms] -
iast 9.258 ms [9.105 ms, 9.41 ms] 4.967 ms (115.8%)
iast_FULL 14.161 ms [13.87 ms, 14.452 ms] 9.87 ms (230.1%)
iast_GLOBAL 10.103 ms [9.93 ms, 10.276 ms] 5.813 ms (135.5%)
profiling 8.967 ms [8.817 ms, 9.117 ms] 4.677 ms (109.0%)
tracing 8.007 ms [7.892 ms, 8.122 ms] 3.717 ms (86.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.334 ms [4.284 ms, 4.384 ms] -
iast 9.128 ms [8.979 ms, 9.277 ms] 4.795 ms (110.6%)
iast_FULL 14.106 ms [13.825 ms, 14.387 ms] 9.772 ms (225.5%)
iast_GLOBAL 10.32 ms [10.141 ms, 10.499 ms] 5.986 ms (138.1%)
profiling 8.997 ms [8.853 ms, 9.141 ms] 4.663 ms (107.6%)
tracing 7.736 ms [7.624 ms, 7.849 ms] 3.403 ms (78.5%)
Request duration reports for petclinic
gantt title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~d160cb7b30, baseline=1.51.0-SNAPSHOT~46d09492c9 dateFormat X axisFormat %s section baseline no_agent (36.903 ms) : 36590, 37216 . : milestone, 36903, appsec (47.407 ms) : 46973, 47841 . : milestone, 47407, code_origins (45.64 ms) : 45242, 46039 . : milestone, 45640, iast (44.097 ms) : 43701, 44493 . : milestone, 44097, profiling (48.905 ms) : 48472, 49338 . : milestone, 48905, tracing (44.076 ms) : 43714, 44437 . : milestone, 44076, section candidate no_agent (36.827 ms) : 36540, 37113 . : milestone, 36827, appsec (46.416 ms) : 45998, 46833 . : milestone, 46416, code_origins (45.749 ms) : 45346, 46151 . : milestone, 45749, iast (43.962 ms) : 43578, 44345 . : milestone, 43962, profiling (47.157 ms) : 46734, 47579 . : milestone, 47157, tracing (43.895 ms) : 43525, 44264 . : milestone, 43895, 
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 36.903 ms [36.59 ms, 37.216 ms] -
appsec 47.407 ms [46.973 ms, 47.841 ms] 10.504 ms (28.5%)
code_origins 45.64 ms [45.242 ms, 46.039 ms] 8.737 ms (23.7%)
iast 44.097 ms [43.701 ms, 44.493 ms] 7.195 ms (19.5%)
profiling 48.905 ms [48.472 ms, 49.338 ms] 12.003 ms (32.5%)
tracing 44.076 ms [43.714 ms, 44.437 ms] 7.173 ms (19.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 36.827 ms [36.54 ms, 37.113 ms] -
appsec 46.416 ms [45.998 ms, 46.833 ms] 9.589 ms (26.0%)
code_origins 45.749 ms [45.346 ms, 46.151 ms] 8.922 ms (24.2%)
iast 43.962 ms [43.578 ms, 44.345 ms] 7.135 ms (19.4%)
profiling 47.157 ms [46.734 ms, 47.579 ms] 10.33 ms (28.1%)
tracing 43.895 ms [43.525 ms, 44.264 ms] 7.068 ms (19.2%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master alexeyk/maven-smoke-test-timeout
git_commit_date 1751046924 1751147372
git_commit_sha 46d0949 d160cb7
release_version 1.51.0-SNAPSHOT~46d09492c9 1.51.0-SNAPSHOT~d160cb7b30
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1751154913 1751154913
ci_job_id 1003861089 1003861089
ci_pipeline_id 69077238 69077238
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zsf25wl1-project-304-concurrent-0-v46bou8n 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zsf25wl1-project-304-concurrent-0-v46bou8n 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat
gantt title tomcat - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~d160cb7b30, baseline=1.51.0-SNAPSHOT~46d09492c9 dateFormat X axisFormat %s section baseline no_agent (1.473 ms) : 1461, 1484 . : milestone, 1473, appsec (2.404 ms) : 2356, 2453 . : milestone, 2404, iast (2.184 ms) : 2123, 2246 . : milestone, 2184, iast_GLOBAL (2.227 ms) : 2166, 2289 . : milestone, 2227, profiling (2.028 ms) : 1979, 2077 . : milestone, 2028, tracing (2.0 ms) : 1953, 2048 . : milestone, 2000, section candidate no_agent (1.477 ms) : 1465, 1488 . : milestone, 1477, appsec (2.396 ms) : 2348, 2445 . : milestone, 2396, iast (2.188 ms) : 2126, 2249 . : milestone, 2188, iast_GLOBAL (2.224 ms) : 2163, 2286 . : milestone, 2224, profiling (2.048 ms) : 1997, 2098 . : milestone, 2048, tracing (2.002 ms) : 1954, 2050 . : milestone, 2002, 
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.473 ms [1.461 ms, 1.484 ms] -
appsec 2.404 ms [2.356 ms, 2.453 ms] 931.605 µs (63.3%)
iast 2.184 ms [2.123 ms, 2.246 ms] 711.632 µs (48.3%)
iast_GLOBAL 2.227 ms [2.166 ms, 2.289 ms] 754.714 µs (51.2%)
profiling 2.028 ms [1.979 ms, 2.077 ms] 555.354 µs (37.7%)
tracing 2.0 ms [1.953 ms, 2.048 ms] 527.452 µs (35.8%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.477 ms [1.465 ms, 1.488 ms] -
appsec 2.396 ms [2.348 ms, 2.445 ms] 919.655 µs (62.3%)
iast 2.188 ms [2.126 ms, 2.249 ms] 710.809 µs (48.1%)
iast_GLOBAL 2.224 ms [2.163 ms, 2.286 ms] 747.626 µs (50.6%)
profiling 2.048 ms [1.997 ms, 2.098 ms] 571.036 µs (38.7%)
tracing 2.002 ms [1.954 ms, 2.05 ms] 525.47 µs (35.6%)
Execution time for biojava
gantt title biojava - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~d160cb7b30, baseline=1.51.0-SNAPSHOT~46d09492c9 dateFormat X axisFormat %s section baseline no_agent (15.091 s) : 15091000, 15091000 . : milestone, 15091000, appsec (15.459 s) : 15459000, 15459000 . : milestone, 15459000, iast (18.229 s) : 18229000, 18229000 . : milestone, 18229000, iast_GLOBAL (17.941 s) : 17941000, 17941000 . : milestone, 17941000, profiling (15.141 s) : 15141000, 15141000 . : milestone, 15141000, tracing (14.998 s) : 14998000, 14998000 . : milestone, 14998000, section candidate no_agent (15.349 s) : 15349000, 15349000 . : milestone, 15349000, appsec (14.756 s) : 14756000, 14756000 . : milestone, 14756000, iast (18.634 s) : 18634000, 18634000 . : milestone, 18634000, iast_GLOBAL (17.986 s) : 17986000, 17986000 . : milestone, 17986000, profiling (15.323 s) : 15323000, 15323000 . : milestone, 15323000, tracing (14.63 s) : 14630000, 14630000 . : milestone, 14630000, 
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.091 s [15.091 s, 15.091 s] -
appsec 15.459 s [15.459 s, 15.459 s] 368.0 ms (2.4%)
iast 18.229 s [18.229 s, 18.229 s] 3.138 s (20.8%)
iast_GLOBAL 17.941 s [17.941 s, 17.941 s] 2.85 s (18.9%)
profiling 15.141 s [15.141 s, 15.141 s] 50.0 ms (0.3%)
tracing 14.998 s [14.998 s, 14.998 s] -93.0 ms (-0.6%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.349 s [15.349 s, 15.349 s] -
appsec 14.756 s [14.756 s, 14.756 s] -593.0 ms (-3.9%)
iast 18.634 s [18.634 s, 18.634 s] 3.285 s (21.4%)
iast_GLOBAL 17.986 s [17.986 s, 17.986 s] 2.637 s (17.2%)
profiling 15.323 s [15.323 s, 15.323 s] -26.0 ms (-0.2%)
tracing 14.63 s [14.63 s, 14.63 s] -719.0 ms (-4.7%)
@AlexeyKuznetsov-DD AlexeyKuznetsov-DD deleted the alexeyk/maven-smoke-test-timeout branch June 30, 2025 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tag: do not merge Do not merge changes

1 participant