Support OpenLineage in spark-3.x-bigquery connectors #1212
Conversation
codelixir commented Apr 15, 2024
codelixir commented - Add openlineage properties to Spark31BigQueryTable class
- Add BigQueryRelationProvider as an abstract class to v2 module, to be extended by BaseBigQuerySource (parent class of all the Spark BigQuery Table Provider classes).
| /gcbrun |
davidrabinowitz left a comment
davidrabinowitz left a comment There was a problem hiding this comment.
Please add an integration test testing that the lineage events are created.
...uery-dsv2-common/src/main/java/com/google/cloud/spark/bigquery/BigQueryRelationProvider.java Outdated Show resolved Hide resolved
...igquery-dsv2-common/src/main/java/com/google/cloud/spark/bigquery/v2/BaseBigQuerySource.java Show resolved Hide resolved
...uery-dsv2-common/src/main/java/com/google/cloud/spark/bigquery/BigQueryRelationProvider.java Show resolved Hide resolved
| /gcbrun |
…gquery-connector-common
| I have moved the logic to the common module, as discussed, so that both dsv1 and dsv2 connectors call the same method internally. |
| /gcbrun |
1 similar comment
| /gcbrun |
| Hey @codelixir, thank you for your contribution! We appreciate your effort. Have you thought about leveraging the |
| Hi @ddebowczyk92 , thanks for the input! We try to keep the DataSource v2 connectors Scala agnostic in order to simplify the usage for customers due to the incompatibility between Scala 2.12 and 2.13. Once this is PR is done, we can think how to incorporate the interface into the connector. |
davidrabinowitz changed the title ...rc/test/java/com/google/cloud/spark/bigquery/integration/OpenLineageIntegrationTestBase.java Show resolved Hide resolved
...rc/test/java/com/google/cloud/spark/bigquery/integration/OpenLineageIntegrationTestBase.java Outdated Show resolved Hide resolved
Signed-off-by: Pahulpreet Singh <pahulpreets@google.com>
| /gcbrun |
...rc/test/java/com/google/cloud/spark/bigquery/integration/OpenLineageIntegrationTestBase.java Outdated Show resolved Hide resolved
Signed-off-by: Pahulpreet Singh <pahulpreets@google.com>
| /gcbrun |
| /gcbrun |
Signed-off-by: Pahulpreet Singh <pahulpreets@google.com>
Signed-off-by: Pahulpreet Singh <pahulpreets@google.com>
Signed-off-by: Pahulpreet Singh <pahulpreets@google.com>
Signed-off-by: Pahulpreet Singh <pahulpreets@google.com> (cherry picked from commit 558f18f)
4 participants
