Skip to content

fix(loader): update shade plugin for spark loader #566

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Feb 24, 2024
Merged

fix(loader): update shade plugin for spark loader #566

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
1a195b3
update shade plugin for spark loader
liuxiaocs7 Jan 6, 2024
cab7443
update license
liuxiaocs7 Jan 6, 2024
178d034
update license
liuxiaocs7 Jan 6, 2024
d144216
update
liuxiaocs7 Jan 6, 2024
File filter

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions hugegraph-dist/release-docs/LICENSE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -321,7 +321,7 @@ See licenses/ for text of these licenses.
(Apache License, Version 2.0) * Gson (com.google.code.gson:gson:2.8.9 - https://github.com/google/gson/gson)
(Apache License, Version 2.0) * Guava InternalFutureFailureAccess and InternalFutures (com.google.guava:failureaccess:1.0.1 - https://github.com/google/guava/failureaccess)
(Apache License, Version 2.0) * Guava ListenableFuture only (com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava - https://github.com/google/guava/listenablefuture)
(Apache License, Version 2.0) * Guava: Google Core Libraries for Java (com.google.guava:guava:32.0.1-jre - https://github.com/google/guava/guava)
(Apache License, Version 2.0) * Guava: Google Core Libraries for Java (com.google.guava:guava:30.0-jre - https://github.com/google/guava/guava)
(Apache License, Version 2.0) * HPPC Collections (com.carrotsearch:hppc:0.7.2 - http://labs.carrotsearch.com/hppc.html/hppc)
(Apache License, Version 2.0) * Hadoop Metrics2 Reporter for Dropwizard Metrics (com.github.joshelser:dropwizard-metrics-hadoop-metrics2-reporter:0.1.2 - https://github.com/joshelser/dropwizard-hadoop-metrics2)
(Apache License, Version 2.0) * Hibernate Validator Engine (org.hibernate.validator:hibernate-validator:6.0.17.Final - http://hibernate.org/validator/hibernate-validator)
Expand All @@ -342,7 +342,7 @@ See licenses/ for text of these licenses.
(Apache License, Version 2.0) * Hive Storage API (org.apache.hive:hive-storage-api:2.7.2 - https://www.apache.org/hive-storage-api/)
(Apache License, Version 2.0) * Hive Upgrade Acid (org.apache.hive:hive-upgrade-acid:3.1.3 - https://www.apache.org/hive-upgrade-acid/)
(Apache License, Version 2.0) * Hive Vector-Code-Gen Utilities (org.apache.hive:hive-vector-code-gen:3.1.3 - https://hive.apache.org/hive-vector-code-gen)
(Apache License, Version 2.0) * J2ObjC Annotations (com.google.j2objc:j2objc-annotations:2.8 - https://github.com/google/j2objc/)
(Apache License, Version 2.0) * J2ObjC Annotations (com.google.j2objc:j2objc-annotations:1.3 - https://github.com/google/j2objc/)
(Apache License, Version 2.0) * JBoss Logging 3 (org.jboss.logging:jboss-logging:3.3.3.Final - http://www.jboss.org)
(Apache License, Version 2.0) * JCIP Annotations under Apache License (com.github.stephenc.jcip:jcip-annotations:1.0-1 - http://stephenc.github.com/jcip-annotations)
(Apache License, Version 2.0) * JPam (net.sf.jpam:jpam:1.1 - http://jpam.sf.net)
Expand Down Expand Up @@ -431,7 +431,7 @@ See licenses/ for text of these licenses.
(Apache License, Version 2.0) * Woodstox (com.fasterxml.woodstox:woodstox-core:5.3.0 - https://github.com/FasterXML/woodstox)
(Apache License, Version 2.0) * aircompressor (io.airlift:aircompressor:0.10 - http://github.com/airlift/aircompressor)
(Apache License, Version 2.0) * aircompressor (io.airlift:aircompressor:0.21 - http://github.com/airlift/aircompressor)
(Apache License, Version 2.0) * error-prone annotations (com.google.errorprone:error_prone_annotations:2.18.0 - http://nexus.sonatype.org/oss-repository-hosting.html/error_prone_parent/error_prone_annotations)
(Apache License, Version 2.0) * error-prone annotations (com.google.errorprone:error_prone_annotations:2.3.4 - http://nexus.sonatype.org/oss-repository-hosting.html/error_prone_parent/error_prone_annotations)
(Apache License, Version 2.0) * htrace-core4 (org.apache.htrace:htrace-core4:4.1.0-incubating - http://incubator.apache.org/projects/htrace.html)
(Apache License, Version 2.0) * htrace-core4 (org.apache.htrace:htrace-core4:4.2.0-incubating - http://incubator.apache.org/projects/htrace.html)
(Apache License, Version 2.0) * jackson-databind (com.fasterxml.jackson.core:jackson-databind:2.12.3 - http://github.com/FasterXML/jackson)
Expand Down Expand Up @@ -582,7 +582,7 @@ Third party MIT licenses
The following components are provided under the MIT License.
See licenses/ for text of these licenses.

(The MIT License) * Checker Qual (org.checkerframework:checker-qual:3.33.0 - https://checkerframework.org)
(The MIT License) * Checker Qual (org.checkerframework:checker-qual:3.5.0 - https://checkerframework.org)
(The MIT License) * JCodings (org.jruby.jcodings:jcodings:1.0.18 - http://nexus.sonatype.org/oss-repository-hosting.html/jcodings)
(The MIT License) * JUL to SLF4J bridge (org.slf4j:jul-to-slf4j:1.7.28 - http://www.slf4j.org)
(The MIT License) * Joni (org.jruby.joni:joni:2.1.11 - http://nexus.sonatype.org/oss-repository-hosting.html/joni)
Expand Down
8 changes: 4 additions & 4 deletions hugegraph-dist/scripts/dependency/known-dependencies.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ audience-annotations-0.5.0.jar
avro-1.10.2.jar
avro-1.7.7.jar
caffeine-2.6.2.jar
checker-qual-3.33.0.jar
checker-qual-3.5.0.jar
classmate-1.4.0.jar
commons-beanutils-1.9.4.jar
commons-cli-1.3.1.jar
Expand Down Expand Up @@ -50,14 +50,14 @@ datanucleus-core-4.1.17.jar
disruptor-3.3.6.jar
dnsjava-2.1.7.jar
dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar
error_prone_annotations-2.18.0.jar
error_prone_annotations-2.3.4.jar
failureaccess-1.0.1.jar
findbugs-annotations-1.3.9-1.jar
flatbuffers-1.2.0-3f79e055.jar
flatbuffers-java-1.9.0.jar
groovy-all-2.4.21.jar
gson-2.8.9.jar
guava-32.0.1-jre.jar
guava-30.0-jre.jar
Copy link
Member

@imbajin imbajin Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could give more conflicts context (maybe paste pic in the comments)
Copy link
Member Author

@liuxiaocs7 liuxiaocs7 Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no guava conflict, but the version in hugegraph-common is 30.0-jre. The guava package does not seem to be used in the toolchain. Can it be inherited directly?
Copy link
Member Author

@liuxiaocs7 liuxiaocs7 Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

image
Copy link
Member

@imbajin imbajin Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no guava conflict, but the version in hugegraph-common is 30.0-jre. The guava package does not seem to be used in the toolchain. Can it be inherited directly?

we do need to use common to manage guava version rather than define it separately (if common use 30.0, just keep up with it)
h2-1.4.199.jar
hadoop-annotations-3.3.1.jar
hadoop-auth-3.3.1.jar
Expand Down Expand Up @@ -119,7 +119,7 @@ httpcore-4.4.12.jar
httpcore-4.4.13.jar
ivy-2.4.0.jar
ivy-2.5.0.jar
j2objc-annotations-2.8.jar
j2objc-annotations-1.3.jar
jackson-annotations-2.12.3.jar
jackson-core-2.12.3.jar
jackson-core-asl-1.9.13.jar
Expand Down
2 changes: 1 addition & 1 deletion hugegraph-loader/assembly/static/bin/hugegraph-spark-loader.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ get_params "$@"
echo "engine_params: $ENGINE_PARAMS"
echo "hugegraph_params: $HUGEGRAPH_PARAMS"

ASSEMBLY_JAR_NAME=$(find "${LIB_DIR}" -name 'apache-hugegraph-loader*.jar')
ASSEMBLY_JAR_NAME=$(find "${LIB_DIR}" -name 'hugegraph-loader*.jar')

DEFAULT_APP_NAME="hugegraph-spark-loader"
APP_NAME=${APP_NAME:-$DEFAULT_APP_NAME}
Expand Down
44 changes: 16 additions & 28 deletions hugegraph-loader/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,10 @@
<groupId>com.fasterxml.jackson.jaxrs</groupId>
<artifactId>*</artifactId>
</exclusion>
<exclusion>
<artifactId>error_prone_annotations</artifactId>
<groupId>com.google.errorprone</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
Expand All @@ -201,10 +205,6 @@
<artifactId>sfm-csv</artifactId>
<version>${sfm.csv.version}</version>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>

<!--hbase -->
<dependency>
Expand All @@ -228,6 +228,10 @@
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-json</artifactId>
</exclusion>
<exclusion>
<artifactId>error_prone_annotations</artifactId>
<groupId>com.google.errorprone</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
Expand Down Expand Up @@ -359,6 +363,10 @@
<groupId>org.apache.commons</groupId>
<artifactId>commons-compress</artifactId>
</exclusion>
<exclusion>
<artifactId>guava</artifactId>
<groupId>com.google.guava</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
Expand Down Expand Up @@ -667,39 +675,19 @@
<createDependencyReducedPom>false</createDependencyReducedPom>
<finalName>${final.name}-shaded</finalName>
<relocations>
<relocation>
<pattern>org.glassfish.jersey</pattern>
<shadedPattern>shaded.org.glassfish.jersey</shadedPattern>
</relocation>
<relocation>
<pattern>jakarta.ws.rs</pattern>
<shadedPattern>shaded.jakarta.ws.rs</shadedPattern>
</relocation>
<relocation>
<pattern>com.google.common.base</pattern>
<shadedPattern>shaded.com.google.common.base</shadedPattern>
</relocation>
<relocation>
<pattern>org.glassfish.hk2</pattern>
<shadedPattern>shaded.org.glassfish.hk2</shadedPattern>
<pattern>okio</pattern>
<shadedPattern>shaded.okio</shadedPattern>
</relocation>
<relocation>
<pattern>org.jvnet.hk2</pattern>
<shadedPattern>shaded.org.jvnet.hk2</shadedPattern>
<pattern>okhttp3</pattern>
<shadedPattern>shaded.okhttp3</shadedPattern>
</relocation>
</relocations>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer">
</transformer>
</transformers>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>module-info.class</exclude>
</excludes>
</filter>
</filters>
</configuration>
</execution>
</executions>
Expand Down
16 changes: 0 additions & 16 deletions ...aph-loader/src/main/resources/META-INF/services/shaded.jakarta.ws.rs.client.ClientBuilder
View file
Open in desktop

This file was deleted.

16 changes: 0 additions & 16 deletions ...in/resources/META-INF/services/shaded.org.glassfish.hk2.extension.ServiceLocatorGenerator
View file
Open in desktop

This file was deleted.

8 changes: 4 additions & 4 deletions hugegraph-tools/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,10 @@
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<artifactId>guava</artifactId>
<groupId>com.google.guava</groupId>
Copy link
Member

@imbajin imbajin Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where do we use it? (in spark-connector?)
Copy link
Member Author

@liuxiaocs7 liuxiaocs7 Jan 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

com.google.j2objc:j2objc-annotations conflict occurred after modifying the guava version to 30.0-jre
</exclusion>
</exclusions>
</dependency>
<dependency>
Expand Down Expand Up @@ -167,10 +171,6 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
</dependency>
</dependencies>

<build>
Expand Down
6 changes: 0 additions & 6 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,6 @@
<commons.net.version>3.9.0</commons.net.version>
<commons.math3.version>3.4.1</commons.math3.version>
<commons.logging.version>1.1.3</commons.logging.version>
<guava.version>32.0.1-jre</guava.version>
<lz4.version>1.4.0</lz4.version>
<jcommand-version>1.72</jcommand-version>
<jackson.version>2.12.3</jackson.version>
Expand Down Expand Up @@ -231,11 +230,6 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>${guava.version}</version>
</dependency>
<dependency>
<groupId>commons-fileupload</groupId>
<artifactId>commons-fileupload</artifactId>
Expand Down