Skip to content

feat: Multiple fixes to make continuous mode work #432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jiangmichaellll merged 14 commits into from
Dec 21, 2020
Merged

feat: Multiple fixes to make continuous mode work #432

jiangmichaellll merged 14 commits into from
Dec 21, 2020

Conversation

@jiangmichaellll
Copy link
Contributor

@jiangmichaellll jiangmichaellll commented Dec 17, 2020

This brings the continuous mode working with non-uber jar solution. Uber jar still won't work.

Mainly:

  1. Lower scala version to 2.11 since it's what's provided in spark 2.4.7 in classpath
  2. Fixes the conversion from java types into spark types
  3. Added maven shade plugin for uber jar generation
@jiangmichaellll jiangmichaellll requested review from a team as code owners December 17, 2020 01:11
@product-auto-label product-auto-label bot added the api: pubsublite Issues related to the googleapis/java-pubsublite API. label Dec 17, 2020
@google-cla google-cla bot added the cla: yes This human has signed the Contributor License Agreement. label Dec 17, 2020
@codecov
Copy link

codecov bot commented Dec 17, 2020 •
edited
Loading

Codecov Report

Merging #432 (ffd15b3) into master (34d8d02) will increase coverage by 0.07%.
The diff coverage is 83.87%.

Impacted file tree graph

@@ Coverage Diff @@ ## master #432 +/- ## ============================================ + Coverage 71.15% 71.22% +0.07%  - Complexity 902 905 +3  ============================================ Files 167 167 Lines 4770 4789 +19 Branches 240 242 +2 ============================================ + Hits 3394 3411 +17  - Misses 1238 1240 +2  Partials 138 138
Impacted Files Coverage Δ Complexity Δ
...cloud/pubsublite/spark/PslCredentialsProvider.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...m/google/cloud/pubsublite/spark/PslDataSource.java 0.00% <0.00%> (ø) 0.00 <0.00> (ø)
...m/google/cloud/pubsublite/spark/PslSparkUtils.java 93.33% <100.00%> (+2.63%) 13.00 <5.00> (+4.00)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 34d8d02...7813ffb. Read the comment docs.
dpcollins-google
dpcollins-google reviewed Dec 18, 2020
View reviewed changes
@jiangmichaellll jiangmichaellll merged commit 9df4ccf into master Dec 21, 2020
@jiangmichaellll jiangmichaellll deleted the jiangmichael-spark-continuous-mode-tuning branch December 21, 2020 18:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: pubsublite Issues related to the googleapis/java-pubsublite API. cla: yes This human has signed the Contributor License Agreement.

2 participants

@jiangmichaellll @dpcollins-google