Skip to content

Automatic triggering of CiVis test environment #9734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Automatic triggering of CiVis test environment #9734

wants to merge 2 commits into from
+122 −2

Conversation

daniel-mohedano
Copy link
Contributor

@daniel-mohedano daniel-mohedano commented Oct 13, 2025 •
edited
Loading

What Does This Do

  • Trigger for Test Environment is now automatic instead of manual:
    • After publish-artifacts-to-s3 finishes, the new check-ci-visibility-label job queries GH API to read the PR labels.
    • If the PR contains the comp: ci visibility tag, the Test Environment job is triggered.
    • Test environment are now also responsible of posting PR comments with the performance overhead per project and tested scenario.

Motivation

We want to improve the Test Optimization developer workflow and more actively monitor performance regressions on our side.

Additional Notes

Current limitations:

  • The run-ci-visibility-test-environment job doesn't appear in the PR job list, likely because it is a trigger job. If it fails it will still fail the default-pipeline

Contributor Checklist

Jira ticket: SDTEST-2768
@daniel-mohedano daniel-mohedano added type: enhancement Enhancements and improvements comp: ci visibility Continuous Integration Visibility comp: tooling Build & Tooling labels Oct 13, 2025
@datadog-official Datadog Official
Copy link

datadog-official bot commented Oct 13, 2025 •
edited by datadog-datadog-prod-us1 bot
Loading

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 59.88% (-0.02%)

View detailed report

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 529c394 | Docs | Was this helpful? Give us feedback!
@pr-commenter
Copy link

pr-commenter bot commented Oct 13, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/test-environment-trigger
git_commit_date 1760520675 1760517543
git_commit_sha ff79486 529c394
release_version 1.55.0-SNAPSHOT~ff79486cac 1.55.0-SNAPSHOT~529c394bfd
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1760529724 1760529724
ci_job_id 1180074969 1180074969
ci_pipeline_id 79357466 79357466
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-qzvi8hzk 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-qzvi8hzk 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 7 unstable metrics.

Startup time reports for insecure-bank 
gantt title insecure-bank - global startup overhead: candidate=1.55.0-SNAPSHOT~529c394bfd, baseline=1.55.0-SNAPSHOT~ff79486cac dateFormat X axisFormat %s section tracing Agent [baseline] (1.02 s) : 0, 1019508 Total [baseline] (8.698 s) : 0, 8698359 Agent [candidate] (1.022 s) : 0, 1022372 Total [candidate] (8.674 s) : 0, 8674269 section iast Agent [baseline] (1.153 s) : 0, 1153036 Total [baseline] (9.338 s) : 0, 9337956 Agent [candidate] (1.153 s) : 0, 1153090 Total [candidate] (9.287 s) : 0, 9286946
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.02 s -
Agent iast 1.153 s 133.528 ms (13.1%)
Total tracing 8.698 s -
Total iast 9.338 s 639.596 ms (7.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.022 s -
Agent iast 1.153 s 130.718 ms (12.8%)
Total tracing 8.674 s -
Total iast 9.287 s 612.678 ms (7.1%) 
gantt title insecure-bank - break down per module: candidate=1.55.0-SNAPSHOT~529c394bfd, baseline=1.55.0-SNAPSHOT~ff79486cac dateFormat X axisFormat %s section tracing crashtracking [baseline] (1.473 ms) : 0, 1473 crashtracking [candidate] (1.463 ms) : 0, 1463 BytebuddyAgent [baseline] (694.196 ms) : 0, 694196 BytebuddyAgent [candidate] (696.832 ms) : 0, 696832 GlobalTracer [baseline] (242.034 ms) : 0, 242034 GlobalTracer [candidate] (242.806 ms) : 0, 242806 AppSec [baseline] (32.394 ms) : 0, 32394 AppSec [candidate] (32.569 ms) : 0, 32569 Debugger [baseline] (6.534 ms) : 0, 6534 Debugger [candidate] (6.44 ms) : 0, 6440 Remote Config [baseline] (734.062 µs) : 0, 734 Remote Config [candidate] (702.307 µs) : 0, 702 Telemetry [baseline] (9.294 ms) : 0, 9294 Telemetry [candidate] (9.33 ms) : 0, 9330 Flare Poller [baseline] (11.671 ms) : 0, 11671 Flare Poller [candidate] (10.912 ms) : 0, 10912 section iast crashtracking [baseline] (1.501 ms) : 0, 1501 crashtracking [candidate] (1.491 ms) : 0, 1491 BytebuddyAgent [baseline] (816.206 ms) : 0, 816206 BytebuddyAgent [candidate] (817.756 ms) : 0, 817756 GlobalTracer [baseline] (231.67 ms) : 0, 231670 GlobalTracer [candidate] (231.279 ms) : 0, 231279 AppSec [baseline] (35.298 ms) : 0, 35298 AppSec [candidate] (34.925 ms) : 0, 34925 Debugger [baseline] (6.214 ms) : 0, 6214 Debugger [candidate] (6.179 ms) : 0, 6179 Remote Config [baseline] (655.617 µs) : 0, 656 Remote Config [candidate] (608.003 µs) : 0, 608 Telemetry [baseline] (8.806 ms) : 0, 8806 Telemetry [candidate] (8.595 ms) : 0, 8595 Flare Poller [baseline] (4.245 ms) : 0, 4245 Flare Poller [candidate] (4.242 ms) : 0, 4242 IAST [baseline] (26.852 ms) : 0, 26852 IAST [candidate] (26.414 ms) : 0, 26414
Loading
Startup time reports for petclinic 
gantt title petclinic - global startup overhead: candidate=1.55.0-SNAPSHOT~529c394bfd, baseline=1.55.0-SNAPSHOT~ff79486cac dateFormat X axisFormat %s section tracing Agent [baseline] (1.02 s) : 0, 1020006 Total [baseline] (10.687 s) : 0, 10687178 Agent [candidate] (1.017 s) : 0, 1017221 Total [candidate] (10.742 s) : 0, 10742289 section appsec Agent [baseline] (1.198 s) : 0, 1198309 Total [baseline] (11.031 s) : 0, 11031392 Agent [candidate] (1.195 s) : 0, 1194740 Total [candidate] (11.032 s) : 0, 11031831 section iast Agent [baseline] (1.15 s) : 0, 1150053 Total [baseline] (11.031 s) : 0, 11031357 Agent [candidate] (1.161 s) : 0, 1161133 Total [candidate] (11.033 s) : 0, 11032563 section profiling Agent [baseline] (1.167 s) : 0, 1167367 Total [baseline] (11.051 s) : 0, 11050834 Agent [candidate] (1.162 s) : 0, 1161585 Total [candidate] (11.072 s) : 0, 11071677
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.02 s -
Agent appsec 1.198 s 178.303 ms (17.5%)
Agent iast 1.15 s 130.047 ms (12.7%)
Agent profiling 1.167 s 147.362 ms (14.4%)
Total tracing 10.687 s -
Total appsec 11.031 s 344.214 ms (3.2%)
Total iast 11.031 s 344.179 ms (3.2%)
Total profiling 11.051 s 363.656 ms (3.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.017 s -
Agent appsec 1.195 s 177.518 ms (17.5%)
Agent iast 1.161 s 143.912 ms (14.1%)
Agent profiling 1.162 s 144.363 ms (14.2%)
Total tracing 10.742 s -
Total appsec 11.032 s 289.542 ms (2.7%)
Total iast 11.033 s 290.274 ms (2.7%)
Total profiling 11.072 s 329.389 ms (3.1%) 
gantt title petclinic - break down per module: candidate=1.55.0-SNAPSHOT~529c394bfd, baseline=1.55.0-SNAPSHOT~ff79486cac dateFormat X axisFormat %s section tracing crashtracking [baseline] (1.467 ms) : 0, 1467 crashtracking [candidate] (1.459 ms) : 0, 1459 BytebuddyAgent [baseline] (693.513 ms) : 0, 693513 BytebuddyAgent [candidate] (693.213 ms) : 0, 693213 GlobalTracer [baseline] (243.238 ms) : 0, 243238 GlobalTracer [candidate] (241.624 ms) : 0, 241624 AppSec [baseline] (32.233 ms) : 0, 32233 AppSec [candidate] (32.292 ms) : 0, 32292 Debugger [baseline] (6.422 ms) : 0, 6422 Debugger [candidate] (6.415 ms) : 0, 6415 Remote Config [baseline] (707.861 µs) : 0, 708 Remote Config [candidate] (708.746 µs) : 0, 709 Telemetry [baseline] (9.344 ms) : 0, 9344 Telemetry [candidate] (9.387 ms) : 0, 9387 Flare Poller [baseline] (11.948 ms) : 0, 11948 Flare Poller [candidate] (10.924 ms) : 0, 10924 section appsec crashtracking [baseline] (1.458 ms) : 0, 1458 crashtracking [candidate] (1.453 ms) : 0, 1453 BytebuddyAgent [baseline] (720.008 ms) : 0, 720008 BytebuddyAgent [candidate] (717.673 ms) : 0, 717673 GlobalTracer [baseline] (235.749 ms) : 0, 235749 GlobalTracer [candidate] (234.497 ms) : 0, 234497 AppSec [baseline] (175.737 ms) : 0, 175737 AppSec [candidate] (176.071 ms) : 0, 176071 Debugger [baseline] (6.194 ms) : 0, 6194 Debugger [candidate] (6.175 ms) : 0, 6175 Remote Config [baseline] (631.566 µs) : 0, 632 Remote Config [candidate] (628.118 µs) : 0, 628 Telemetry [baseline] (8.525 ms) : 0, 8525 Telemetry [candidate] (8.414 ms) : 0, 8414 Flare Poller [baseline] (3.928 ms) : 0, 3928 Flare Poller [candidate] (3.886 ms) : 0, 3886 IAST [baseline] (24.933 ms) : 0, 24933 IAST [candidate] (24.849 ms) : 0, 24849 section iast crashtracking [baseline] (1.467 ms) : 0, 1467 crashtracking [candidate] (1.465 ms) : 0, 1465 BytebuddyAgent [baseline] (814.255 ms) : 0, 814255 BytebuddyAgent [candidate] (823.213 ms) : 0, 823213 GlobalTracer [baseline] (231.345 ms) : 0, 231345 GlobalTracer [candidate] (232.736 ms) : 0, 232736 AppSec [baseline] (35.339 ms) : 0, 35339 AppSec [candidate] (35.379 ms) : 0, 35379 Debugger [baseline] (6.134 ms) : 0, 6134 Debugger [candidate] (6.229 ms) : 0, 6229 Remote Config [baseline] (608.737 µs) : 0, 609 Remote Config [candidate] (632.846 µs) : 0, 633 Telemetry [baseline] (8.721 ms) : 0, 8721 Telemetry [candidate] (8.667 ms) : 0, 8667 Flare Poller [baseline] (4.182 ms) : 0, 4182 Flare Poller [candidate] (4.324 ms) : 0, 4324 IAST [baseline] (26.52 ms) : 0, 26520 IAST [candidate] (26.751 ms) : 0, 26751 section profiling ProfilingAgent [baseline] (108.92 ms) : 0, 108920 ProfilingAgent [candidate] (108.441 ms) : 0, 108441 crashtracking [baseline] (1.452 ms) : 0, 1452 crashtracking [candidate] (1.433 ms) : 0, 1433 BytebuddyAgent [baseline] (724.779 ms) : 0, 724779 BytebuddyAgent [candidate] (720.67 ms) : 0, 720670 GlobalTracer [baseline] (218.559 ms) : 0, 218559 GlobalTracer [candidate] (217.786 ms) : 0, 217786 AppSec [baseline] (32.731 ms) : 0, 32731 AppSec [candidate] (32.506 ms) : 0, 32506 Debugger [baseline] (7.37 ms) : 0, 7370 Debugger [candidate] (6.456 ms) : 0, 6456 Remote Config [baseline] (792.613 µs) : 0, 793 Remote Config [candidate] (822.635 µs) : 0, 823 Telemetry [baseline] (15.202 ms) : 0, 15202 Telemetry [candidate] (16.147 ms) : 0, 16147 Flare Poller [baseline] (4.159 ms) : 0, 4159 Flare Poller [candidate] (4.208 ms) : 0, 4208 Profiling [baseline] (110.072 ms) : 0, 110072 Profiling [candidate] (109.582 ms) : 0, 109582
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/test-environment-trigger
git_commit_date 1760520675 1760517543
git_commit_sha ff79486 529c394
release_version 1.55.0-SNAPSHOT~ff79486cac 1.55.0-SNAPSHOT~529c394bfd
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1760529449 1760529449
ci_job_id 1180074970 1180074970
ci_pipeline_id 79357466 79357466
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-q6em4tzf 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-q6em4tzf 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 2 performance improvements and 3 performance regressions! Performance is the same for 7 metrics, 12 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:insecure-bank:no_agent:high_load worse
[+140.433µs; +246.011µs] or [+3.285%; +5.754%]		 unstable
[-161.648op/s; +70.898op/s] or [-15.064%; +6.607%]		 4.468ms 1027.688op/s 4.275ms 1073.062op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load better
[-947.230µs; -538.138µs] or [-8.554%; -4.860%]		 unstable
[-21.841op/s; +81.841op/s] or [-5.200%; +19.486%]		 10.331ms 450.000op/s 11.074ms 420.000op/s
scenario:load:petclinic:iast:high_load better
[-2.977ms; -2.135ms] or [-6.547%; -4.697%]		 unstable
[-1.342op/s; +13.567op/s] or [-1.303%; +13.179%]		 42.910ms 109.050op/s 45.466ms 102.938op/s
scenario:load:petclinic:profiling:high_load worse
[+1.775ms; +2.723ms] or [+3.741%; +5.739%]		 unstable
[-11.110op/s; +2.160op/s] or [-11.259%; +2.189%]		 49.701ms 94.200op/s 47.452ms 98.675op/s
scenario:load:petclinic:tracing:high_load worse
[+2.096ms; +2.914ms] or [+4.863%; +6.761%]		 unstable
[-13.276op/s; +1.326op/s] or [-12.227%; +1.221%]		 45.602ms 102.600op/s 43.097ms 108.575op/s
Request duration reports for insecure-bank 
gantt title insecure-bank - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~529c394bfd, baseline=1.55.0-SNAPSHOT~ff79486cac dateFormat X axisFormat %s section baseline no_agent (4.275 ms) : 4227, 4323 . : milestone, 4275, iast (9.569 ms) : 9412, 9726 . : milestone, 9569, iast_FULL (14.175 ms) : 13896, 14454 . : milestone, 14175, iast_GLOBAL (11.074 ms) : 10875, 11273 . : milestone, 11074, profiling (9.05 ms) : 8899, 9201 . : milestone, 9050, tracing (7.81 ms) : 7697, 7923 . : milestone, 7810, section candidate no_agent (4.468 ms) : 4418, 4519 . : milestone, 4468, iast (9.38 ms) : 9226, 9533 . : milestone, 9380, iast_FULL (14.166 ms) : 13887, 14446 . : milestone, 14166, iast_GLOBAL (10.331 ms) : 10151, 10512 . : milestone, 10331, profiling (8.978 ms) : 8838, 9118 . : milestone, 8978, tracing (7.846 ms) : 7734, 7958 . : milestone, 7846,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.275 ms [4.227 ms, 4.323 ms] -
iast 9.569 ms [9.412 ms, 9.726 ms] 5.294 ms (123.8%)
iast_FULL 14.175 ms [13.896 ms, 14.454 ms] 9.9 ms (231.6%)
iast_GLOBAL 11.074 ms [10.875 ms, 11.273 ms] 6.799 ms (159.0%)
profiling 9.05 ms [8.899 ms, 9.201 ms] 4.775 ms (111.7%)
tracing 7.81 ms [7.697 ms, 7.923 ms] 3.535 ms (82.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.468 ms [4.418 ms, 4.519 ms] -
iast 9.38 ms [9.226 ms, 9.533 ms] 4.911 ms (109.9%)
iast_FULL 14.166 ms [13.887 ms, 14.446 ms] 9.698 ms (217.0%)
iast_GLOBAL 10.331 ms [10.151 ms, 10.512 ms] 5.863 ms (131.2%)
profiling 8.978 ms [8.838 ms, 9.118 ms] 4.51 ms (100.9%)
tracing 7.846 ms [7.734 ms, 7.958 ms] 3.378 ms (75.6%)
Request duration reports for petclinic 
gantt title petclinic - request duration [CI 0.99] : candidate=1.55.0-SNAPSHOT~529c394bfd, baseline=1.55.0-SNAPSHOT~ff79486cac dateFormat X axisFormat %s section baseline no_agent (36.183 ms) : 35879, 36486 . : milestone, 36183, appsec (49.078 ms) : 48632, 49524 . : milestone, 49078, code_origins (43.671 ms) : 43295, 44047 . : milestone, 43671, iast (45.466 ms) : 45068, 45864 . : milestone, 45466, profiling (47.452 ms) : 47025, 47878 . : milestone, 47452, tracing (43.097 ms) : 42733, 43462 . : milestone, 43097, section candidate no_agent (36.672 ms) : 36382, 36962 . : milestone, 36672, appsec (48.823 ms) : 48380, 49266 . : milestone, 48823, code_origins (44.313 ms) : 43927, 44698 . : milestone, 44313, iast (42.91 ms) : 42527, 43293 . : milestone, 42910, profiling (49.701 ms) : 49247, 50155 . : milestone, 49701, tracing (45.602 ms) : 45208, 45997 . : milestone, 45602,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 36.183 ms [35.879 ms, 36.486 ms] -
appsec 49.078 ms [48.632 ms, 49.524 ms] 12.895 ms (35.6%)
code_origins 43.671 ms [43.295 ms, 44.047 ms] 7.489 ms (20.7%)
iast 45.466 ms [45.068 ms, 45.864 ms] 9.283 ms (25.7%)
profiling 47.452 ms [47.025 ms, 47.878 ms] 11.269 ms (31.1%)
tracing 43.097 ms [42.733 ms, 43.462 ms] 6.915 ms (19.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 36.672 ms [36.382 ms, 36.962 ms] -
appsec 48.823 ms [48.38 ms, 49.266 ms] 12.151 ms (33.1%)
code_origins 44.313 ms [43.927 ms, 44.698 ms] 7.641 ms (20.8%)
iast 42.91 ms [42.527 ms, 43.293 ms] 6.238 ms (17.0%)
profiling 49.701 ms [49.247 ms, 50.155 ms] 13.029 ms (35.5%)
tracing 45.602 ms [45.208 ms, 45.997 ms] 8.93 ms (24.4%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master daniel.mohedano/test-environment-trigger
git_commit_date 1760520675 1760517543
git_commit_sha ff79486 529c394
release_version 1.55.0-SNAPSHOT~ff79486cac 1.55.0-SNAPSHOT~529c394bfd
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1760529949 1760529949
ci_job_id 1180074971 1180074971
ci_pipeline_id 79357466 79357466
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-pt84quid 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-pt84quid 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 2 unstable metrics.

Execution time for biojava 
gantt title biojava - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~529c394bfd, baseline=1.55.0-SNAPSHOT~ff79486cac dateFormat X axisFormat %s section baseline no_agent (14.909 s) : 14909000, 14909000 . : milestone, 14909000, appsec (15.035 s) : 15035000, 15035000 . : milestone, 15035000, iast (18.28 s) : 18280000, 18280000 . : milestone, 18280000, iast_GLOBAL (18.035 s) : 18035000, 18035000 . : milestone, 18035000, profiling (15.42 s) : 15420000, 15420000 . : milestone, 15420000, tracing (15.104 s) : 15104000, 15104000 . : milestone, 15104000, section candidate no_agent (14.706 s) : 14706000, 14706000 . : milestone, 14706000, appsec (14.918 s) : 14918000, 14918000 . : milestone, 14918000, iast (18.791 s) : 18791000, 18791000 . : milestone, 18791000, iast_GLOBAL (18.204 s) : 18204000, 18204000 . : milestone, 18204000, profiling (14.942 s) : 14942000, 14942000 . : milestone, 14942000, tracing (14.961 s) : 14961000, 14961000 . : milestone, 14961000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.909 s [14.909 s, 14.909 s] -
appsec 15.035 s [15.035 s, 15.035 s] 126.0 ms (0.8%)
iast 18.28 s [18.28 s, 18.28 s] 3.371 s (22.6%)
iast_GLOBAL 18.035 s [18.035 s, 18.035 s] 3.126 s (21.0%)
profiling 15.42 s [15.42 s, 15.42 s] 511.0 ms (3.4%)
tracing 15.104 s [15.104 s, 15.104 s] 195.0 ms (1.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.706 s [14.706 s, 14.706 s] -
appsec 14.918 s [14.918 s, 14.918 s] 212.0 ms (1.4%)
iast 18.791 s [18.791 s, 18.791 s] 4.085 s (27.8%)
iast_GLOBAL 18.204 s [18.204 s, 18.204 s] 3.498 s (23.8%)
profiling 14.942 s [14.942 s, 14.942 s] 236.0 ms (1.6%)
tracing 14.961 s [14.961 s, 14.961 s] 255.0 ms (1.7%)
Execution time for tomcat 
gantt title tomcat - execution time [CI 0.99] : candidate=1.55.0-SNAPSHOT~529c394bfd, baseline=1.55.0-SNAPSHOT~ff79486cac dateFormat X axisFormat %s section baseline no_agent (1.482 ms) : 1470, 1493 . : milestone, 1482, appsec (3.75 ms) : 3530, 3970 . : milestone, 3750, iast (2.225 ms) : 2161, 2289 . : milestone, 2225, iast_GLOBAL (2.278 ms) : 2213, 2343 . : milestone, 2278, profiling (2.479 ms) : 2325, 2634 . : milestone, 2479, tracing (2.043 ms) : 1992, 2093 . : milestone, 2043, section candidate no_agent (1.48 ms) : 1468, 1491 . : milestone, 1480, appsec (3.724 ms) : 3503, 3945 . : milestone, 3724, iast (2.237 ms) : 2173, 2302 . : milestone, 2237, iast_GLOBAL (2.264 ms) : 2199, 2329 . : milestone, 2264, profiling (2.064 ms) : 2011, 2116 . : milestone, 2064, tracing (2.042 ms) : 1992, 2093 . : milestone, 2042,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.482 ms [1.47 ms, 1.493 ms] -
appsec 3.75 ms [3.53 ms, 3.97 ms] 2.268 ms (153.1%)
iast 2.225 ms [2.161 ms, 2.289 ms] 743.551 µs (50.2%)
iast_GLOBAL 2.278 ms [2.213 ms, 2.343 ms] 796.208 µs (53.7%)
profiling 2.479 ms [2.325 ms, 2.634 ms] 997.623 µs (67.3%)
tracing 2.043 ms [1.992 ms, 2.093 ms] 561.023 µs (37.9%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.48 ms [1.468 ms, 1.491 ms] -
appsec 3.724 ms [3.503 ms, 3.945 ms] 2.245 ms (151.7%)
iast 2.237 ms [2.173 ms, 2.302 ms] 757.7 µs (51.2%)
iast_GLOBAL 2.264 ms [2.199 ms, 2.329 ms] 784.535 µs (53.0%)
profiling 2.064 ms [2.011 ms, 2.116 ms] 584.138 µs (39.5%)
tracing 2.042 ms [1.992 ms, 2.093 ms] 562.282 µs (38.0%)
@daniel-mohedano daniel-mohedano added the tag: no release notes Changes to exclude from release notes label Oct 13, 2025
@daniel-mohedano daniel-mohedano marked this pull request as ready for review October 13, 2025 10:29
@daniel-mohedano daniel-mohedano requested a review from a team as a code owner October 13, 2025 10:29
@daniel-mohedano daniel-mohedano requested review from erikayasuda and removed request for a team October 13, 2025 10:29
PerfectSlayer
PerfectSlayer reviewed Oct 13, 2025
View reviewed changes
.gitlab/ci-visibility-tests.yml
echo "Found PR #${PR_NUMBER}"
# Check if PR has the CI visibility label
if pr_has_label "$PR_NUMBER" "comp: ci visibility"; then
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❔ question: ‏What about using run-tests: label category?
As I will soon enforce to only have one comp: or inst: label to prevent duplicate changelog, I would recommend using the labels dedicated to CI task run.
Copy link
Contributor

@nikita-tkachenko-datadog nikita-tkachenko-datadog Oct 13, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

then we'll have to add run-tests:... to every one of our PRs just for the sake of it. Adding comp:ci-visibility is something we already do and won't forget to do.

I will soon enforce to only have one comp: or inst:

That's a pity, is there any other way we could avoid duplicates? E.g. if there's an instrumentation label, use it to determine the section, and then fallback to the comp one?
@DataDog DataDog deleted a comment from cit-pr-commenter bot Oct 15, 2025
@cit-pr-commenter cit-pr-commenter
Copy link

Test Environment - sbt-scalatest

Job Status: success

Scenario Overhead (%)
agent 55.28
agentEvpProxy 55.72
@cit-pr-commenter cit-pr-commenter
Copy link

Test Environment - nebula-release-plugin

Job Status: success

Scenario Overhead (%)
agent 37.87
agentless 36.72
agentlessCodeCoverage 42.89
agentlessLineCoverage 73.32
@cit-pr-commenter cit-pr-commenter
Copy link

Test Environment - pass4s

Job Status: success

Scenario Overhead (%)
agent 14.70
agentless 8.24
agentlessCodeCoverage 13.14
@cit-pr-commenter cit-pr-commenter
Copy link

Test Environment - jolokia

Job Status: success

Scenario Overhead (%)
agent 90.54
agentless 85.18
agentlessCodeCoverage 95.70
agentlessLineCoverage 96.51
@cit-pr-commenter cit-pr-commenter
Copy link

Test Environment - okhttp

Job Status: success

Scenario Overhead (%)
agent 15.69
agentless 16.22
agentlessCodeCoverage 19.99
agentlessLineCoverage 35.23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp: ci visibility Continuous Integration Visibility comp: tooling Build & Tooling tag: no release notes Changes to exclude from release notes type: enhancement Enhancements and improvements

3 participants

@daniel-mohedano @PerfectSlayer @nikita-tkachenko-datadog