Skip to content

fix(loader): update shade plugin for spark loader #566

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 4 commits into from
Feb 24, 2024
Merged

fix(loader): update shade plugin for spark loader #566

merged 4 commits into from
Feb 24, 2024

Conversation

@liuxiaocs7
Copy link
Member

@liuxiaocs7 liuxiaocs7 commented Jan 6, 2024 •
edited
Loading

Purpose of the PR

Main Changes

Verifying these changes

  • Trivial rework / code cleanup without any test coverage. (No Need)
  • Already covered by existing tests, such as (please modify tests here).
  • Need tests and can be verified as follows:
    • xxx

Does this PR potentially affect the following parts?

  • Nope
  • Dependencies (add/update license info)
  • Modify configurations
  • The public API
  • Other affects (typed here)

Documentation Status

  • Doc - TODO
  • Doc - Done
  • Doc - No Need
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working java Pull requests that update Java code loader hugegraph-loader labels Jan 6, 2024
@github-actions github-actions bot added the tools hugegraph-tools label Jan 6, 2024
@liuxiaocs7
Copy link
Member Author

liuxiaocs7 commented Jan 6, 2024 •
edited
Loading

exec

./bin/hugegraph-spark-loader.sh --master local[*] --name spark-hugegraph-loader --file ./example/spark/struct.json --host 192.168.110.214 --port 8080 --graph hugegraph

load data successfully.
@codecov
Copy link

codecov bot commented Jan 6, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (b066b80) 62.49% compared to head (d144216) 62.55%.
Report is 19 commits behind head on master.

Additional details and impacted files 
@@ Coverage Diff @@ ## master #566 +/- ## ============================================ + Coverage 62.49% 62.55% +0.05%  - Complexity 1903 1923 +20  ============================================ Files 262 267 +5 Lines 9541 9677 +136 Branches 886 888 +2 ============================================ + Hits 5963 6053 +90  - Misses 3190 3235 +45  - Partials 388 389 +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.
@dosubot dosubot bot added size:M This PR changes 30-99 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jan 6, 2024
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:M This PR changes 30-99 lines, ignoring generated files. labels Jan 6, 2024
imbajin
imbajin reviewed Jan 8, 2024
View reviewed changes
Copy link
Member

@imbajin imbajin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

currently in the release-1.2.0, we can't load data well in spark-connector?
hugegraph-tools/pom.xml
</exclusion>
<exclusion>
<artifactId>guava</artifactId>
<groupId>com.google.guava</groupId>
Copy link
Member

@imbajin imbajin Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where do we use it? (in spark-connector?)
Copy link
Member Author

@liuxiaocs7 liuxiaocs7 Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

com.google.j2objc:j2objc-annotations conflict occurred after modifying the guava version to 30.0-jre
hugegraph-dist/scripts/dependency/known-dependencies.txt
groovy-all-2.4.21.jar
gson-2.8.9.jar
guava-32.0.1-jre.jar
guava-30.0-jre.jar
Copy link
Member

@imbajin imbajin Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could give more conflicts context (maybe paste pic in the comments)
Copy link
Member Author

@liuxiaocs7 liuxiaocs7 Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no guava conflict, but the version in hugegraph-common is 30.0-jre. The guava package does not seem to be used in the toolchain. Can it be inherited directly?
Copy link
Member Author

@liuxiaocs7 liuxiaocs7 Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image
Copy link
Member

@imbajin imbajin Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no guava conflict, but the version in hugegraph-common is 30.0-jre. The guava package does not seem to be used in the toolchain. Can it be inherited directly?

we do need to use common to manage guava version rather than define it separately (if common use 30.0, just keep up with it)
@liuxiaocs7
Copy link
Member Author

liuxiaocs7 commented Jan 8, 2024

currently in the release-1.2.0, we can't load data well in spark-connector?

no, in loader/HugeGraphSparkLoader
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Feb 20, 2024
@imbajin
Copy link
Member

imbajin commented Feb 20, 2024

Note: we need basic test & more doc for the spark-loader & spark-connector (users may feel confused about them now)
@javeme javeme merged commit 26d6b31 into apache:master Feb 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working java Pull requests that update Java code lgtm This PR has been approved by a maintainer loader hugegraph-loader size:L This PR changes 100-499 lines, ignoring generated files. tools hugegraph-tools

3 participants

@liuxiaocs7 @imbajin @javeme