Configure OpenLineage if present in Spark instrumentation #8541
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
| Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
BenchmarksStartupParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 65 metrics, 6 unstable metrics. Startup time reports for insecure-bankgantt title insecure-bank - global startup overhead: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed dateFormat X axisFormat %s section tracing Agent [baseline] (1.057 s) : 0, 1056860 Total [baseline] (8.756 s) : 0, 8756395 Agent [candidate] (1.052 s) : 0, 1052036 Total [candidate] (8.669 s) : 0, 8668550 section iast Agent [baseline] (1.179 s) : 0, 1179159 Total [baseline] (9.25 s) : 0, 9249768 Agent [candidate] (1.18 s) : 0, 1180295 Total [candidate] (9.241 s) : 0, 9240989 section iast_HARDCODED_SECRET_DISABLED Agent [baseline] (1.18 s) : 0, 1179974 Total [baseline] (9.193 s) : 0, 9193198 Agent [candidate] (1.189 s) : 0, 1189252 Total [candidate] (9.257 s) : 0, 9257106 section iast_TELEMETRY_OFF Agent [baseline] (1.175 s) : 0, 1175267 Total [baseline] (9.225 s) : 0, 9224944 Agent [candidate] (1.18 s) : 0, 1180245 Total [candidate] (9.235 s) : 0, 9234735
gantt title insecure-bank - break down per module: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed dateFormat X axisFormat %s section tracing BytebuddyAgent [baseline] (722.977 ms) : 0, 722977 BytebuddyAgent [candidate] (719.423 ms) : 0, 719423 GlobalTracer [baseline] (241.072 ms) : 0, 241072 GlobalTracer [candidate] (239.602 ms) : 0, 239602 AppSec [baseline] (54.787 ms) : 0, 54787 AppSec [candidate] (54.277 ms) : 0, 54277 Debugger [baseline] (4.466 ms) : 0, 4466 Debugger [candidate] (4.411 ms) : 0, 4411 Remote Config [baseline] (695.648 µs) : 0, 696 Remote Config [candidate] (698.178 µs) : 0, 698 Telemetry [baseline] (12.221 ms) : 0, 12221 Telemetry [candidate] (13.046 ms) : 0, 13046 section iast BytebuddyAgent [baseline] (837.265 ms) : 0, 837265 BytebuddyAgent [candidate] (838.061 ms) : 0, 838061 GlobalTracer [baseline] (229.65 ms) : 0, 229650 GlobalTracer [candidate] (229.073 ms) : 0, 229073 IAST [baseline] (22.626 ms) : 0, 22626 IAST [candidate] (23.535 ms) : 0, 23535 AppSec [baseline] (55.637 ms) : 0, 55637 AppSec [candidate] (55.685 ms) : 0, 55685 Debugger [baseline] (4.134 ms) : 0, 4134 Debugger [candidate] (4.173 ms) : 0, 4173 Remote Config [baseline] (609.876 µs) : 0, 610 Remote Config [candidate] (610.992 µs) : 0, 611 Telemetry [baseline] (8.752 ms) : 0, 8752 Telemetry [candidate] (8.808 ms) : 0, 8808 section iast_HARDCODED_SECRET_DISABLED BytebuddyAgent [baseline] (837.669 ms) : 0, 837669 BytebuddyAgent [candidate] (845.656 ms) : 0, 845656 GlobalTracer [baseline] (229.522 ms) : 0, 229522 GlobalTracer [candidate] (230.526 ms) : 0, 230526 IAST [baseline] (22.735 ms) : 0, 22735 IAST [candidate] (22.921 ms) : 0, 22921 AppSec [baseline] (56.071 ms) : 0, 56071 AppSec [candidate] (56.053 ms) : 0, 56053 Debugger [baseline] (4.159 ms) : 0, 4159 Debugger [candidate] (4.134 ms) : 0, 4134 Remote Config [baseline] (619.782 µs) : 0, 620 Remote Config [candidate] (608.445 µs) : 0, 608 Telemetry [baseline] (8.746 ms) : 0, 8746 Telemetry [candidate] (8.735 ms) : 0, 8735 section iast_TELEMETRY_OFF BytebuddyAgent [baseline] (833.838 ms) : 0, 833838 BytebuddyAgent [candidate] (839.013 ms) : 0, 839013 GlobalTracer [baseline] (229.238 ms) : 0, 229238 GlobalTracer [candidate] (229.025 ms) : 0, 229025 IAST [baseline] (22.287 ms) : 0, 22287 IAST [candidate] (22.331 ms) : 0, 22331 AppSec [baseline] (56.115 ms) : 0, 56115 AppSec [candidate] (55.92 ms) : 0, 55920 Debugger [baseline] (4.188 ms) : 0, 4188 Debugger [candidate] (4.163 ms) : 0, 4163 Remote Config [baseline] (616.429 µs) : 0, 616 Remote Config [candidate] (617.675 µs) : 0, 618 Telemetry [baseline] (8.65 ms) : 0, 8650 Telemetry [candidate] (8.664 ms) : 0, 8664 Startup time reports for petclinicgantt title petclinic - global startup overhead: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed dateFormat X axisFormat %s section tracing Agent [baseline] (1.049 s) : 0, 1049442 Total [baseline] (10.53 s) : 0, 10530093 Agent [candidate] (1.052 s) : 0, 1052209 Total [candidate] (10.461 s) : 0, 10461334 section appsec Agent [baseline] (1.194 s) : 0, 1194103 Total [baseline] (10.722 s) : 0, 10722065 Agent [candidate] (1.194 s) : 0, 1193507 Total [candidate] (10.727 s) : 0, 10726572 section iast Agent [baseline] (1.188 s) : 0, 1187776 Total [baseline] (11.07 s) : 0, 11070306 Agent [candidate] (1.18 s) : 0, 1179747 Total [candidate] (11.041 s) : 0, 11040520 section profiling Agent [baseline] (1.273 s) : 0, 1272804 Total [baseline] (10.863 s) : 0, 10862747 Agent [candidate] (1.283 s) : 0, 1283341 Total [candidate] (10.905 s) : 0, 10905045
gantt title petclinic - break down per module: candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed dateFormat X axisFormat %s section tracing BytebuddyAgent [baseline] (717.637 ms) : 0, 717637 BytebuddyAgent [candidate] (718.733 ms) : 0, 718733 GlobalTracer [baseline] (238.937 ms) : 0, 238937 GlobalTracer [candidate] (239.567 ms) : 0, 239567 AppSec [baseline] (54.411 ms) : 0, 54411 AppSec [candidate] (54.482 ms) : 0, 54482 Debugger [baseline] (5.153 ms) : 0, 5153 Debugger [candidate] (4.433 ms) : 0, 4433 Remote Config [baseline] (685.957 µs) : 0, 686 Remote Config [candidate] (701.371 µs) : 0, 701 Telemetry [baseline] (12.132 ms) : 0, 12132 Telemetry [candidate] (13.817 ms) : 0, 13817 section appsec BytebuddyAgent [baseline] (738.003 ms) : 0, 738003 BytebuddyAgent [candidate] (737.564 ms) : 0, 737564 GlobalTracer [baseline] (235.92 ms) : 0, 235920 GlobalTracer [candidate] (235.483 ms) : 0, 235483 AppSec [baseline] (175.841 ms) : 0, 175841 AppSec [candidate] (175.629 ms) : 0, 175629 Debugger [baseline] (4.279 ms) : 0, 4279 Debugger [candidate] (4.294 ms) : 0, 4294 Remote Config [baseline] (640.17 µs) : 0, 640 Remote Config [candidate] (640.948 µs) : 0, 641 Telemetry [baseline] (8.204 ms) : 0, 8204 Telemetry [candidate] (8.541 ms) : 0, 8541 IAST [baseline] (21.415 ms) : 0, 21415 IAST [candidate] (21.708 ms) : 0, 21708 section iast BytebuddyAgent [baseline] (844.286 ms) : 0, 844286 BytebuddyAgent [candidate] (837.964 ms) : 0, 837964 GlobalTracer [baseline] (230.656 ms) : 0, 230656 GlobalTracer [candidate] (229.547 ms) : 0, 229547 AppSec [baseline] (55.641 ms) : 0, 55641 AppSec [candidate] (55.889 ms) : 0, 55889 Debugger [baseline] (4.162 ms) : 0, 4162 Debugger [candidate] (4.109 ms) : 0, 4109 Remote Config [baseline] (622.265 µs) : 0, 622 Remote Config [candidate] (615.666 µs) : 0, 616 Telemetry [baseline] (8.795 ms) : 0, 8795 Telemetry [candidate] (8.625 ms) : 0, 8625 IAST [baseline] (22.99 ms) : 0, 22990 IAST [candidate] (22.555 ms) : 0, 22555 section profiling ProfilingAgent [baseline] (102.23 ms) : 0, 102230 ProfilingAgent [candidate] (103.317 ms) : 0, 103317 BytebuddyAgent [baseline] (709.029 ms) : 0, 709029 BytebuddyAgent [candidate] (715.025 ms) : 0, 715025 GlobalTracer [baseline] (348.306 ms) : 0, 348306 GlobalTracer [candidate] (351.262 ms) : 0, 351262 AppSec [baseline] (53.761 ms) : 0, 53761 AppSec [candidate] (53.734 ms) : 0, 53734 Debugger [baseline] (4.252 ms) : 0, 4252 Debugger [candidate] (4.297 ms) : 0, 4297 Remote Config [baseline] (694.487 µs) : 0, 694 Remote Config [candidate] (707.246 µs) : 0, 707 Telemetry [baseline] (8.93 ms) : 0, 8930 Telemetry [candidate] (9.033 ms) : 0, 9033 Profiling [baseline] (102.256 ms) : 0, 102256 Profiling [candidate] (103.342 ms) : 0, 103342 LoadParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 18 unstable metrics. Request duration reports for insecure-bankgantt title insecure-bank - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed dateFormat X axisFormat %s section baseline no_agent (386.813 µs) : 367, 406 . : milestone, 387, iast (516.4 µs) : 495, 538 . : milestone, 516, iast_FULL (735.829 µs) : 714, 758 . : milestone, 736, iast_GLOBAL (573.642 µs) : 550, 597 . : milestone, 574, iast_HARDCODED_SECRET_DISABLED (511.929 µs) : 490, 533 . : milestone, 512, iast_INACTIVE (467.099 µs) : 445, 489 . : milestone, 467, iast_TELEMETRY_OFF (508.127 µs) : 486, 530 . : milestone, 508, tracing (470.344 µs) : 447, 493 . : milestone, 470, section candidate no_agent (376.262 µs) : 357, 396 . : milestone, 376, iast (514.304 µs) : 493, 536 . : milestone, 514, iast_FULL (737.11 µs) : 715, 759 . : milestone, 737, iast_GLOBAL (560.891 µs) : 539, 583 . : milestone, 561, iast_HARDCODED_SECRET_DISABLED (515.283 µs) : 493, 537 . : milestone, 515, iast_INACTIVE (468.495 µs) : 447, 490 . : milestone, 468, iast_TELEMETRY_OFF (500.898 µs) : 479, 522 . : milestone, 501, tracing (462.803 µs) : 442, 484 . : milestone, 463,
Request duration reports for petclinicgantt title petclinic - request duration [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed dateFormat X axisFormat %s section baseline no_agent (1.359 ms) : 1339, 1379 . : milestone, 1359, appsec (1.743 ms) : 1719, 1767 . : milestone, 1743, appsec_no_iast (1.748 ms) : 1725, 1772 . : milestone, 1748, code_origins (1.698 ms) : 1672, 1724 . : milestone, 1698, iast (1.524 ms) : 1499, 1549 . : milestone, 1524, profiling (1.548 ms) : 1523, 1573 . : milestone, 1548, tracing (1.508 ms) : 1483, 1534 . : milestone, 1508, section candidate no_agent (1.346 ms) : 1326, 1366 . : milestone, 1346, appsec (1.736 ms) : 1712, 1760 . : milestone, 1736, appsec_no_iast (1.757 ms) : 1732, 1782 . : milestone, 1757, code_origins (1.685 ms) : 1657, 1712 . : milestone, 1685, iast (1.525 ms) : 1501, 1550 . : milestone, 1525, profiling (1.565 ms) : 1539, 1590 . : milestone, 1565, tracing (1.508 ms) : 1483, 1533 . : milestone, 1508,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics. Execution time for biojavagantt title biojava - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed dateFormat X axisFormat %s section baseline no_agent (15.454 s) : 15454000, 15454000 . : milestone, 15454000, appsec (15.176 s) : 15176000, 15176000 . : milestone, 15176000, iast (18.992 s) : 18992000, 18992000 . : milestone, 18992000, iast_GLOBAL (17.605 s) : 17605000, 17605000 . : milestone, 17605000, profiling (15.549 s) : 15549000, 15549000 . : milestone, 15549000, tracing (14.945 s) : 14945000, 14945000 . : milestone, 14945000, section candidate no_agent (15.421 s) : 15421000, 15421000 . : milestone, 15421000, appsec (14.938 s) : 14938000, 14938000 . : milestone, 14938000, iast (18.812 s) : 18812000, 18812000 . : milestone, 18812000, iast_GLOBAL (18.259 s) : 18259000, 18259000 . : milestone, 18259000, profiling (15.206 s) : 15206000, 15206000 . : milestone, 15206000, tracing (14.78 s) : 14780000, 14780000 . : milestone, 14780000,
Execution time for tomcatgantt title tomcat - execution time [CI 0.99] : candidate=1.48.0-SNAPSHOT~83cf093182, baseline=1.48.0-SNAPSHOT~32046a39ed dateFormat X axisFormat %s section baseline no_agent (1.472 ms) : 1460, 1483 . : milestone, 1472, appsec (2.371 ms) : 2324, 2418 . : milestone, 2371, iast (2.166 ms) : 2106, 2226 . : milestone, 2166, iast_GLOBAL (2.201 ms) : 2141, 2260 . : milestone, 2201, profiling (2.046 ms) : 1998, 2095 . : milestone, 2046, tracing (1.981 ms) : 1935, 2027 . : milestone, 1981, section candidate no_agent (1.47 ms) : 1458, 1481 . : milestone, 1470, appsec (2.373 ms) : 2327, 2420 . : milestone, 2373, iast (2.153 ms) : 2094, 2213 . : milestone, 2153, iast_GLOBAL (2.19 ms) : 2131, 2249 . : milestone, 2190, profiling (2.013 ms) : 1965, 2062 . : milestone, 2013, tracing (1.985 ms) : 1939, 2030 . : milestone, 1985,
|
harels reviewed Mar 13, 2025
...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java Outdated Show resolved Hide resolved
mobuchowski force-pushed the mobuchowski/openlineage-agent-direct branch from d36e910 to d48bb91 Compare 
kr-igor approved these changes Mar 13, 2025
| "_[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$", ""); | ||
| } | ||
| | ||
| private Optional<Class> loadClass(String className) { |
There was a problem hiding this comment.
Why don't we just instrument OL listener instead? I.e. add listener setup code which sets all required settings if not present?
| private final int MAX_COLLECTION_SIZE = 5000; | ||
| private final int MAX_ACCUMULATOR_SIZE = 50000; | ||
| private final String RUNTIME_TAGS_PREFIX = "spark.datadog.tags."; | ||
| private static final String AGENT_OL_ENDPOINT = "openlineage/api/v1/lineage"; |
There was a problem hiding this comment.
Not directly related to this PR since it is an agent change, but to keep the same default as openlineage, wondering if it could be easier to keep the endpoint name as /api/v1/lineage in the agent
There was a problem hiding this comment.
My idea was that /api/v1/lineage endpoint was too confusing, and that as we're setting this - not customer - the additional endpoint setting does not matter that much. We can do a change on the client still, it was not yet released.
| finishApplication(System.currentTimeMillis(), null, 0, null); | ||
| } | ||
| })); | ||
| initApplicationSpanIfNotInitialized(); |
There was a problem hiding this comment.
I don't think we should create the application span in the constructor/onApplicationStart.
In the case of Databricks or streaming jobs, we are currently not creating a application span:
- in databricks the parent of a spark job is the databricks task that launched it
- in streaming, we don't attach the streaming span to the application span
In those cases, I think we should skip overriding OpenLineage for now
There was a problem hiding this comment.
Removed this here, added check for databricks/streaming
mobuchowski force-pushed the mobuchowski/openlineage-agent-direct branch 5 times, most recently from c3e99e1 to a582374 Compare mobuchowski force-pushed the mobuchowski/openlineage-agent-direct branch 6 times, most recently from 55d9b0d to ccd94f3 Compare mobuchowski force-pushed the mobuchowski/openlineage-agent-direct branch from b4637a8 to 1c43c96 Compare 
amarziali reviewed Mar 27, 2025
| public static void exit(@Advice.This Object self) throws IllegalAccessException { | ||
| LoggerFactory.getLogger(Config.class).debug("Checking for OpenLineageSparkListener"); | ||
| try { | ||
| Field conf = self.getClass().getDeclaredField("conf"); |
There was a problem hiding this comment.
can you use @FieldValue to have the same without using the reflection?
There was a problem hiding this comment.
I would like to - but the instrumented class clones the object so it's not the same one as what's passed to the method.
There was a problem hiding this comment.
I was not proposing to capture the argument of the constructor but to use @FieldValue to directly map the field (https://javadoc.io/doc/net.bytebuddy/byte-buddy/1.10.2/net/bytebuddy/asm/Advice.FieldValue.html) like this:
public static void exit(@Advice.This Object self, @FieldValue("conf") SparkConf conf) { ... amarziali reviewed Mar 27, 2025
| @Override | ||
| public String[] knownMatchingTypes() { | ||
| return new String[] { | ||
| "io.openlineage.spark.agent.OpenLineageSparkListener", "org.apache.spark.util.Utils" |
There was a problem hiding this comment.
why org.apache.spark.util.Utils is also matching the advice activation while the method advice is only filtering by io.openlineage.spark.agent.OpenLineageSparkListener?
amarziali reviewed Mar 27, 2025
...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractSparkInstrumentation.java Outdated Show resolved Hide resolved
amarziali reviewed Mar 27, 2025
...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractSparkInstrumentation.java Outdated Show resolved Hide resolved
amarziali reviewed Mar 27, 2025
...on/spark/src/main/java/datadog/trace/instrumentation/spark/AbstractDatadogSparkListener.java Outdated Show resolved Hide resolved
amarziali reviewed Mar 27, 2025
...rk/spark_2.12/src/main/java/datadog/trace/instrumentation/spark/Spark212Instrumentation.java Outdated Show resolved Hide resolved
amarziali reviewed Mar 27, 2025
...rk/spark_2.13/src/main/java/datadog/trace/instrumentation/spark/Spark213Instrumentation.java Outdated Show resolved Hide resolved
amarziali requested changes Mar 27, 2025
mobuchowski force-pushed the mobuchowski/openlineage-agent-direct branch from 1c43c96 to 34d6cb4 Compare 
| Thanks @amarziali, applied the changes. Added one more change - using |
mobuchowski force-pushed the mobuchowski/openlineage-agent-direct branch from 34d6cb4 to 5cd3c14 Compare amarziali approved these changes Mar 31, 2025
mobuchowski force-pushed the mobuchowski/openlineage-agent-direct branch from 5cd3c14 to f596261 Compare 7 tasks
mobuchowski force-pushed the mobuchowski/openlineage-agent-direct branch 3 times, most recently from bdb681c to c34a423 Compare 
Signed-off-by: Maciej Obuchowski <maciej.obuchowski@datadoghq.com>
Signed-off-by: Maciej Obuchowski <maciej.obuchowski@datadoghq.com>
mobuchowski force-pushed the mobuchowski/openlineage-agent-direct branch from c34a423 to 83cf093 Compare 
svc-squareup-copybara pushed a commit to cashapp/misk that referenced this pull request Apr 11, 2025
| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | org.flywaydb.flyway | plugin | misk/gradle/libs.versions.toml | gradle | minor | `11.6.0` -> `11.7.0` | | [com.squareup.okio:okio-fakefilesystem](https://github.com/square/okio) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `3.10.2` -> `3.11.0` | | [com.squareup.okio:okio](https://github.com/square/okio) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `3.10.2` -> `3.11.0` | | [com.autonomousapps.dependency-analysis](https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin) | plugin | misk/gradle/libs.versions.toml | gradle | minor | `2.15.0` -> `2.16.0` | | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.47.3` -> `1.48.1` | | [com.datadoghq:dd-trace-ot](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.47.3` -> `1.48.1` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.18` -> `2.31.20` | --- ### Release Notes <details> <summary>square/okio (com.squareup.okio:okio-fakefilesystem)</summary> ### [`v3.11.0`](https://github.com/square/okio/blob/HEAD/CHANGELOG.md#Version-3110) *2025-04-09* - Fix: Clear the deflater's byte array reference - New: Faster implementation of `String.decodeHex()` on Kotlin/JS. - New: Declare `EXACTLY_ONCE` execution for blocks like `Closeable.use {}` and `FileSystem.read {}`. - Upgrade: \[Kotlin 2.1.20]\[kotlin\_2\_1\_20]. </details> <details> <summary>autonomousapps/dependency-analysis-android-gradle-plugin (com.autonomousapps.dependency-analysis)</summary> ### [`v2.16.0`](https://github.com/autonomousapps/dependency-analysis-android-gradle-plugin/blob/HEAD/CHANGELOG.md#Version-2160) - \[Feat]: support `com.android.test` projects. - \[Feat]: support typesafe project accessors with opt-in. ```kotlin dependencyAnalysis { useTypesafeProjectAccessors(true) // false by default } ``` </details> <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.48.1`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.48.1): 1.48.1 ### Components #### Tracer internal logging - 🐛 Remove print line causing unnecessary logs ([#​8687](DataDog/dd-trace-java#8687) - [@​sarahchen6](https://github.com/sarahchen6)) ### [`v1.48.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.48.0): 1.48.0 ### Known Bugs > \[!NOTE] > If you are experiencing issues with spamming timeout logs, please update to the [latest version](https://github.com/DataDog/dd-trace-java/releases/latest) or set [JDK_SOCKET_ENABLED](https://github.com/DataDog/dd-trace-java/blob/33fc3c9a9b7cda3beda88b8b3e5224ae2b10764a/dd-trace-api/src/main/java/datadog/trace/api/config/GeneralConfig.java#L98) to false. ### Components #### Application Security Management (IAST) - ✨ Fix vulnerability location org.jose4j.lang.HashUtil ([#​8610](DataDog/dd-trace-java#8610) - [@​jandro996](https://github.com/jandro996)) - ✨ Fix weak randomness in oracle.ucp.util.OpaqueString ([#​8609](DataDog/dd-trace-java#8609) - [@​jandro996](https://github.com/jandro996)) - ✨ Fix weak hash false positive in oracle.security.o5logon.O5Logon ([#​8608](DataDog/dd-trace-java#8608) - [@​jandro996](https://github.com/jandro996)) - 🐛 Prevent before callsites targeting constructors in super calls ([#​8549](DataDog/dd-trace-java#8549) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### Application Security Management (WAF) - ✨ Update login events public SDK to V2 ([#​8620](DataDog/dd-trace-java#8620) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - 🐛 Send RASP LFI capability only when AppSec is statically enabled ([#​8573](DataDog/dd-trace-java#8573) - [@​jandro996](https://github.com/jandro996)) - ✨ Improve detection of missing request end events ([#​8510](DataDog/dd-trace-java#8510) - [@​smola](https://github.com/smola)) - 🧹 Remove remote configuration for API Security sampling rate ([#​8486](DataDog/dd-trace-java#8486) - [@​smola](https://github.com/smola)) - ✨ Add setUser to user monitoring SDK ([#​8482](DataDog/dd-trace-java#8482) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Add missing address for signup event ([#​8469](DataDog/dd-trace-java#8469) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Allow login events SDK to be used with appsec disabled ([#​8464](DataDog/dd-trace-java#8464) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Add support for endpoint discovery in spring mvc ([#​8352](DataDog/dd-trace-java#8352) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ New API Security sampling algorithm ([#​8178](DataDog/dd-trace-java#8178) - [@​ValentinZakharov](https://github.com/ValentinZakharov)) #### Build & Tooling - ✨ Add buffer size customizability to JDK UDS support ([#​8629](DataDog/dd-trace-java#8629) - [@​sarahchen6](https://github.com/sarahchen6)) - ✨ Add JDK built-in support for UDS on Java 16+ ([#​8314](DataDog/dd-trace-java#8314) - [@​sarahchen6](https://github.com/sarahchen6)) #### Configuration at Runtime - 🐛 Send RASP LFI capability only when AppSec is statically enabled ([#​8573](DataDog/dd-trace-java#8573) - [@​jandro996](https://github.com/jandro996)) #### Continuous Integration Visibility - 🐛 Prevent double reporting of Scalatest events when using SBT with test forking ([#​8682](DataDog/dd-trace-java#8682) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Shutdown CI Visibility test event handlers before tracer ([#​8677](DataDog/dd-trace-java#8677) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Do not apply JUnit 4 instrumentation to MUnit runners ([#​8675](DataDog/dd-trace-java#8675), [#​8683](DataDog/dd-trace-java#8683) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Remove error log when source path resolution fails on isModified check ([#​8663](DataDog/dd-trace-java#8663) - [@​daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Implement tests reordering for JUnit 4 ([#​8650](DataDog/dd-trace-java#8650) - [@​daniel-mohedano](https://github.com/daniel-mohedano)) - 🐛 Set default Attempt to Fix retries if none provided from the backend ([#​8615](DataDog/dd-trace-java#8615) - [@​daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Allow to manually set PR info ([#​8566](DataDog/dd-trace-java#8566) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - 🐛 Fix Test Optimization init when repo root cannot be determined ([#​8533](DataDog/dd-trace-java#8533) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add capabilities tagging ([#​8499](DataDog/dd-trace-java#8499), [#​8540](DataDog/dd-trace-java#8540) - [@​daniel-mohedano](https://github.com/daniel-mohedano)) #### Crash tracking - 🐛 Remove dependency on bash from crash/oome uploder scripts ([#​8652](DataDog/dd-trace-java#8652) - [@​jbachorik](https://github.com/jbachorik)) #### Data Streams Monitoring - ✨ e2e pipeline configuration when data jobs is enabled ([#​8553](DataDog/dd-trace-java#8553) - [@​kr-igor](https://github.com/kr-igor)) #### Dynamic Instrumentation - 🐛 Fix In-Product when config is empty ([#​8679](DataDog/dd-trace-java#8679) - [@​jpbempel](https://github.com/jpbempel)) - ✨ Add support for filtering shaded third-party libs ([#​8612](DataDog/dd-trace-java#8612) - [@​jpbempel](https://github.com/jpbempel)) - ✨ Add In-Product Enablement ([#​8587](DataDog/dd-trace-java#8587) - [@​jpbempel](https://github.com/jpbempel)) - ✨⚡ Reduce footprint of SourceFile tracking ([#​8524](DataDog/dd-trace-java#8524) - [@​jpbempel](https://github.com/jpbempel)) - ✨⚡ Optimize the SourceFile tracking ([#​8520](DataDog/dd-trace-java#8520) - [@​jpbempel](https://github.com/jpbempel)) #### OpenTracing - 🧹 Remove activeScope() use in OpenTracing shim ([#​8478](DataDog/dd-trace-java#8478) - [@​mcculls](https://github.com/mcculls)) #### Profiling - ✨ Add profiler env check command to AgentCLI ([#​8671](DataDog/dd-trace-java#8671) - [@​jbachorik](https://github.com/jbachorik)) - ✨ Bump ddprof to 1.23.0 ([#​8668](DataDog/dd-trace-java#8668) - [@​jbachorik](https://github.com/jbachorik)) - Fix a crash related to ElfParser::loadSymbolTable ([#​191](DataDog/dd-trace-java#191)) by [@​yanglong1010](https://github.com/yanglong1010) in DataDog/java-profiler#192 - Unwind String.indexOf intrinsic on AArch64 by [@​MattAlp](https://github.com/MattAlp) in DataDog/java-profiler#193 - Fix Java 24 support by [@​jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#194 - A set of fixes related to clang, aarch64 and musl pecularities of vmstructs stack unwinder by [@​jbachorik](https://github.com/jbachorik) in DataDog/java-profiler#199 - 🐛 Remove process information from JFR recording ([#​8661](DataDog/dd-trace-java#8661) - [@​r1viollet](https://github.com/r1viollet)) - 🐛 Make TempLocationManager USER aware ([#​8605](DataDog/dd-trace-java#8605) - [@​jbachorik](https://github.com/jbachorik)) - ✨ Extract git tags from embedded git.properties and datadog_git.properties ([#​8561](DataDog/dd-trace-java#8561) - [@​wmouchere](https://github.com/wmouchere)) #### Telemetry - 🐛 Fix appsec.rasp.error and appsec.waf.error telemetry metrics ([#​8624](DataDog/dd-trace-java#8624) - [@​jandro996](https://github.com/jandro996)) - ✨ Create metric: appsec.rasp.rule.skipped ([#​8618](DataDog/dd-trace-java#8618) - [@​jandro996](https://github.com/jandro996)) - ✨ Extract git tags from embedded git.properties and datadog_git.properties ([#​8561](DataDog/dd-trace-java#8561) - [@​wmouchere](https://github.com/wmouchere)) #### Testing - 🧹 Simplify ssi tests one-pipeline ([#​8558](DataDog/dd-trace-java#8558) - [@​robertomonteromiguel](https://github.com/robertomonteromiguel)) - ✨ Add smoke tests for java's concurrent API ([#​8438](DataDog/dd-trace-java#8438) - [@​sarahchen6](https://github.com/sarahchen6)) #### Trace context propagation - ✨ Adding Support for `TRACE_PROPAGATION_BEHAVIOR_EXTRACT` ([#​8535](DataDog/dd-trace-java#8535) - [@​mhlidd](https://github.com/mhlidd)) #### Tracer core - 🐛 Ensure shaded helpers have unique names ([#​8559](DataDog/dd-trace-java#8559) - [@​amarziali](https://github.com/amarziali)) - ✨ Support common config sources for user-provided git info ([#​8547](DataDog/dd-trace-java#8547) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Make the default config sources more robust when a security manager is installed ([#​8544](DataDog/dd-trace-java#8544) - [@​mcculls](https://github.com/mcculls)) - ✨ Support targeting services with configurations in stable configuration file ([#​8526](DataDog/dd-trace-java#8526) - [@​mtoffl01](https://github.com/mtoffl01)) - ✨ Add new parser for `DD_TAGS` and prioritizing `DD_SERVICE` ([#​8296](DataDog/dd-trace-java#8296) - [@​mhlidd](https://github.com/mhlidd)) #### Tracer internal logging - 🐛 Add missing debug log for the cloudPayloadTaggingServices config ([#​8600](DataDog/dd-trace-java#8600) - [@​ygree](https://github.com/ygree)) - ✨ Add the possibility to output the logs of the Java tracer in JSON ([#​8083](DataDog/dd-trace-java#8083) - [@​cecile75](https://github.com/cecile75)) #### Tracer public API - ✨ Introducing `DD_TRACE_EXPERIMENTAL_FEATURES_ENABLED` Config ([#​8536](DataDog/dd-trace-java#8536) - [@​mhlidd](https://github.com/mhlidd)) - ✨ Config Consistency Round 2 ([#​8489](DataDog/dd-trace-java#8489) - [@​mhlidd](https://github.com/mhlidd)) ### Instrumentations #### - 🐛 Fix NPE in getMdcCopy of LoggingEventInstrumentation ([#​8599](DataDog/dd-trace-java#8599) - [@​ygree](https://github.com/ygree)) #### Apache Spark instrumentation - ✨ Instrument Runtime.exit() to finish spark application spans ([#​8572](DataDog/dd-trace-java#8572) - [@​paul-laffon-dd](https://github.com/paul-laffon-dd)) - ✨ Configure OpenLineage if present in Spark instrumentation ([#​8541](DataDog/dd-trace-java#8541) - [@​mobuchowski](https://github.com/mobuchowski)) #### Armeria Instrumentation - ✨ Support armeria grpc 1.32.3 ([#​8606](DataDog/dd-trace-java#8606) - [@​github-actions](https://github.com/github-actions)\[bot]) #### AWS DynamoDB Instrumentation - ✨ Create DynamoDB instrumentation + add span pointers for `updateItem` and `deleteItem` ([#​8490](DataDog/dd-trace-java#8490) - [@​nhulston](https://github.com/nhulston)) #### AWS SDK instrumentation - ✨ Add DynamoDB in DEFAULT_TRACE_CLOUD_PAYLOAD_TAGGING_SERVICES ([#​8595](DataDog/dd-trace-java#8595) - [@​joeyzhao2018](https://github.com/joeyzhao2018)) #### Azure Functions instrumentation - ✨ Enable tracer computed trace metrics by default for Azure Functions ([#​8518](DataDog/dd-trace-java#8518) - [@​duncanpharvey](https://github.com/duncanpharvey)) - 💡 Add azure-functions instrumentation ([#​8432](DataDog/dd-trace-java#8432) - [@​duncanpharvey](https://github.com/duncanpharvey)) #### Core Java language instrumentation - 🐛 Fix ForkJoinPool.execute() instrumentation on Java 21+ ([#​8560](DataDog/dd-trace-java#8560) - [@​PerfectSlayer](https://github.com/PerfectSlayer)) #### Eclipse Vert.x instrumentation - ✨ Add vertx postgresql client instrumentation ([#​8471](DataDog/dd-trace-java#8471) - [@​vandonr](https://github.com/vandonr) - thanks for the contribution!) #### Kafka instrumentation - ✨ Support and test kafka-clients 4 ([#​8581](DataDog/dd-trace-java#8581) - [@​amarziali](https://github.com/amarziali)) #### Kotlin instrumentation - ✨ Avoid disconnected traces when using Kotlin flowOn ([#​8651](DataDog/dd-trace-java#8651) - [@​mcculls](https://github.com/mcculls)) #### OpenTelemetry instrumentation - 🧹 Migrate OtelContext wrapper to new internal Context API ([#​8645](DataDog/dd-trace-java#8645) - [@​mcculls](https://github.com/mcculls)) #### Spring instrumentation - 🐛 Support CompletableFuture on spring webmvc controllers ([#​8659](DataDog/dd-trace-java#8659) - [@​amarziali](https://github.com/amarziali)) - ✨ Add support for endpoint discovery in spring mvc ([#​8352](DataDog/dd-trace-java#8352) - [@​manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### WebSocket Instrumentation - ✨ Instrument Jetty websocket pojo ([#​8562](DataDog/dd-trace-java#8562) - [@​amarziali](https://github.com/amarziali)) - 💡 Instrument Java Websocket API (JSR356) ([#​8440](DataDog/dd-trace-java#8440) - [@​amarziali](https://github.com/amarziali)) #### All other instrumentations - ✨ Introduce cache for peer.hostname lookup ([#​8601](DataDog/dd-trace-java#8601) - [@​mcculls](https://github.com/mcculls)) - ✨ Support pekko http 1.1 ([#​8532](DataDog/dd-trace-java#8532) - [@​amarziali](https://github.com/amarziali)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 331314f71acaced3adc75ea5d7e855c248d593fc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
What Does This Do
If OpenLineage library is present on the classpath, instrument SparkListenerBus to inject the OpenLineageSparkListener with additional tags ourselves.
Motivation
https://docs.google.com/document/d/14nmf3UcqhzoOfooiveoEBUO-HHejKalLR-qAKzbR2PI/edit?tab=t.0#heading=h.dpfdi5vyt85h
https://docs.google.com/document/d/1EwNCsnhnilL6YHYMg4UYwqK_yKgBEUsxys669h8l_-M/edit?tab=t.0#heading=h.o8z5ox632qxx
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]