feat: support docker for loader #530

aroundabout · 2023-11-06T08:01:57Z

Purpose of the PR

support docker for hugegraph-loader

Main Changes

Dockerfile for loader
an example docker-compose for loader

usage

(add the usage to README and doc later)

current test image is dandelionivy/loader

cd hugegraph-loader/docker/example/ docker-compose up -d docker exec -it loader bash

sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h graph -p 8080

It should be like:

user@LAPTOP-MRGBSP9S:~/incubator-hugegraph-toolchain/hugegraph-loader/docker/example$ sudo docker-compose up -d Creating graph ... done Creating loader ... done Creating hubble ... done user@LAPTOP-MRGBSP9S:~/incubator-hugegraph-toolchain/hugegraph-loader/docker/example$ sudo docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 59f431e622d0 hugegraph/hubble "./bin/start-hubble.…" About a minute ago Up About a minute 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp hubble 55360d18f7d7 dandelionivy/loader "/usr/bin/dumb-init …" About a minute ago Up About a minute loader 35b3feefc8bf hugegraph/hugegraph "/usr/bin/dumb-init …" About a minute ago Up About a minute 0.0.0.0:18081->8080/tcp, :::18081->8080/tcp graph user@LAPTOP-MRGBSP9S:~/incubator-hugegraph-toolchain/hugegraph-loader/docker/example$ sudo docker exec -it loader bash root@55360d18f7d7:/loader# sh bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -h graph -p 8080 ... vertices/edges loaded this time : 8/6 -------------------------------------------------- count metrics input read success : 14 input read failure : 0 vertex parse success : 8 vertex parse failure : 0 vertex insert success : 8 vertex insert failure : 0 edge parse success : 6 edge parse failure : 0 edge insert success : 6 edge insert failure : 0 -------------------------------------------------- meter metrics total time : 0.199s read time : 0.046s load time : 0.153s vertex load time : 0.077s vertex load rate(vertices/s) : 103 edge load time : 0.112s edge load rate(edges/s) : 53 root@55360d18f7d7:/loader#

Then we can see the result in hubble:

Verifying these changes

Trivial rework / code cleanup without any test coverage. (No Need)
Already covered by existing tests, such as (please modify tests here).
Need tests and can be verified as follows:
- my test docker image is here dandelionivy/loader

Does this PR potentially affect the following parts?

Nope
Dependencies (add/update license info)
Modify configurations
The public API
Other affects (typed here)

Documentation Status

Doc - TODO
Doc - Done
Doc - No Need

codecov · 2023-11-06T08:17:10Z

Codecov Report

Merging #530 (56ac508) into master (71e623d) will decrease coverage by 0.03%.
Report is 1 commits behind head on master.
The diff coverage is n/a.

@@ Coverage Diff @@ ## master #530 +/- ## ============================================ - Coverage 62.49% 62.47% -0.03%  + Complexity 1903 930 -973  ============================================ Files 262 93 -169 Lines 9541 4509 -5032 Branches 886 529 -357 ============================================ - Hits 5963 2817 -3146  + Misses 3190 1483 -1707  + Partials 388 209 -179

see 169 files with indirect coverage changes

📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!

aroundabout · 2023-11-06T08:54:05Z

hugegraph-loader/Dockerfile

+WORKDIR /pkg
+
+RUN set -x \
+ && mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp


It seems that in the center maven repo, the client-1.0.0 does not have some methods like istext(). Hence, we should install the jar locally.

It seems that in the center maven repo, the client-1.0.0 does not have some methods like istext(). Hence, we should install the jar locally.

seems we don't rely on fixed version? we use $revision in the project, so install client first is necessary (if we want build the latest image)

hugegraph-loader/Dockerfile

imbajin · 2023-11-07T02:18:13Z

@liuxiaocs7 could also check the PR when free

imbajin · 2023-11-07T02:21:12Z

hugegraph-loader/Dockerfile

+WORKDIR /pkg
+
+RUN set -x \
+ && mvn install -pl hugegraph-client,hugegraph-loader -am -Dmaven.javadoc.skip=true -DskipTests -ntp


It seems that in the center maven repo, the client-1.0.0 does not have some methods like istext(). Hence, we should install the jar locally.

seems we don't rely on fixed version? we use $revision in the project, so install client first is necessary (if we want build the latest image)

imbajin · 2023-11-07T02:22:28Z

hugegraph-loader/Dockerfile

+ENTRYPOINT ["/usr/bin/dumb-init", "--"]
+CMD [ "/bin/bash" ]


maybe check other usage way for dumb-init ?

It seems there is no special usage. We only need to add dumb-init in entrypoint. Maybe tail -f /dev/null is better than /bin/bash.

imbajin · 2023-11-07T02:23:03Z

hugegraph-loader/README.md

-## Building
+## 2. Usage for Docker(Recommand)
+
+- deploy `loader` with Docker


Suggested change

- deploy `loader` with Docker

- run `loader` with Docker

imbajin · 2023-11-07T02:25:08Z

hugegraph-loader/README.md

+#### 2.2.1 enter the docker container
+
+```bash
+docker exec -it loader bash


remember update here when doc update apache/incubator-hugegraph-doc#299 (comment)

imbajin · 2023-11-07T02:28:41Z

hugegraph-loader/README.md

consider also add a new section (run with docker) in root (README) https://github.com/apache/incubator-hugegraph-toolchain/blob/master/README.md#modules

and keep the module section clear

liuxiaocs7 · 2023-11-10T08:39:14Z

Hi, @aroundabout, thanks for your outstanding efforts on this issue, i test locally and find some issues and hope your help.

Firstly, i try to build the image locally but fails.

(base) kirin@kirinbiotech:~/lx/incubator-hugegraph-toolchain$ docker build -t hg-loader:1.0 hugegraph-loader/

Then, i use loader image from here: https://hub.docker.com/r/dandelionivy/loader/tags where server and hubble deployed separately.

kirin@kirinbiotech:~/lx/incubator-hugegraph-toolchain/hugegraph-loader/assembly/static$ docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -p 8080 -h 192.168.110.158

but still fails to load data.

CC: @imbajin

aroundabout · 2023-11-10T08:44:09Z

Hi, @aroundabout, thanks for your outstanding efforts on this issue, i test locally and find some issues.

Firstly, i try to build the image locally but fails.
(base) kirin@kirinbiotech:~/lx/incubator-hugegraph-toolchain$ docker build -t hg-loader:1.0 hugegraph-loader/ 
Then, i use loader image from here: https://hub.docker.com/r/dandelionivy/loader/tags
kirin@kirinbiotech:~/lx/incubator-hugegraph-toolchain/hugegraph-loader/assembly/static$ docker exec -it loader bin/hugegraph-loader.sh -g hugegraph -f example/file/struct.json -s example/file/schema.groovy -p 8080 -h 192.168.110.158 
but still fails to load data.

How to build locally
try docker build -f hugegraph-loader/Dockerfile hg-loader .
Because loader depends on client and the client-1.0.0 in maven repo does not have some methods like 'istext()', so we should build client locally in dockerfile. Hence, the docker build context should be root path
The question is caused by server [Bug] Unexpected stream error with gzip & need enhance filter logic incubator-hugegraph#2346. The PR to fix it has been opened fix(api): refactor/downgrade record logic for slow log incubator-hugegraph#2347

liuxiaocs7 · 2023-11-10T08:52:22Z

How to build locally
try docker build -f hugegraph-loader/Dockerfile hg-loader .
Because loader depends on client and the client-1.0.0 in maven repo does not have some methods like 'istext()', so we should build client locally in dockerfile. Hence, the docker build context should be root path

Thanks for your details explain, i see.

docker build -f hugegraph-loader/Dockerfile hg-loader .

seems fails too? And does it need to be added to the readme?

aroundabout · 2023-11-10T08:59:27Z

How to build locally
try docker build -f hugegraph-loader/Dockerfile hg-loader .
Because loader depends on client and the client-1.0.0 in maven repo does not have some methods like 'istext()', so we should build client locally in dockerfile. Hence, the docker build context should be root path

Thanks for your details explain, i see.
docker build -f hugegraph-loader/Dockerfile hg-loader . 
seems fails too? And does it need to be added to the readme?

Sorry, I forgot an argument. XD

docker build -f hugegraph-loader/Dockerfile -t hg-loader .

It can works now.

liuxiaocs7 · 2023-11-10T09:03:54Z

How to build locally
try docker build -f hugegraph-loader/Dockerfile hg-loader .
Because loader depends on client and the client-1.0.0 in maven repo does not have some methods like 'istext()', so we should build client locally in dockerfile. Hence, the docker build context should be root path

Thanks for your details explain, i see.
docker build -f hugegraph-loader/Dockerfile hg-loader . 
seems fails too? And does it need to be added to the readme?
Sorry, I forgot an argument. XD

docker build -f hugegraph-loader/Dockerfile -t hg-loader .

It can works now.

works!

liuxiaocs7

LGTM~

imbajin

THX

feat: support docker for loader

d6aff34

github-actions bot added the loader hugegraph-loader label Nov 6, 2023

add empty line at end of file

2d70c48

supplement license

0ae8ced

aroundabout commented Nov 6, 2023

View reviewed changes

imbajin reviewed Nov 6, 2023

View reviewed changes

hugegraph-loader/Dockerfile Show resolved Hide resolved

supplement readme

9fa0614

github-actions bot added the hubble hugegraph-hubble label Nov 6, 2023

supplement readme

705b66f

imbajin reviewed Nov 7, 2023

View reviewed changes

aroundabout added 2 commits November 8, 2023 12:57

change dockerfile cmd and readme

acc9b54

opt doc and docker compose

56ac508

liuxiaocs7 approved these changes Nov 13, 2023

View reviewed changes

imbajin approved these changes Nov 13, 2023

View reviewed changes

simon824 approved these changes Nov 13, 2023

View reviewed changes

simon824 merged commit e8fcdd3 into apache:master Nov 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: support docker for loader #530

feat: support docker for loader #530

Uh oh!

aroundabout commented Nov 6, 2023 •

edited

Loading

codecov bot commented Nov 6, 2023 •

edited

Loading

aroundabout Nov 6, 2023

imbajin Nov 7, 2023

Uh oh!

imbajin commented Nov 7, 2023

imbajin Nov 7, 2023

imbajin Nov 7, 2023

aroundabout Nov 7, 2023

imbajin Nov 7, 2023

imbajin Nov 7, 2023

imbajin Nov 7, 2023

liuxiaocs7 commented Nov 10, 2023 •

edited

Loading

aroundabout commented Nov 10, 2023 •

edited

Loading

liuxiaocs7 commented Nov 10, 2023

aroundabout commented Nov 10, 2023

liuxiaocs7 commented Nov 10, 2023

liuxiaocs7 left a comment

imbajin left a comment

Labels

4 participants

feat: support docker for loader #530

feat: support docker for loader #530

Uh oh!

Conversation

aroundabout commented Nov 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of the PR

Main Changes

usage

Verifying these changes

Does this PR potentially affect the following parts?

Documentation Status

codecov bot commented Nov 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

imbajin commented Nov 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liuxiaocs7 commented Nov 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

aroundabout commented Nov 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

liuxiaocs7 commented Nov 10, 2023

aroundabout commented Nov 10, 2023

liuxiaocs7 commented Nov 10, 2023

liuxiaocs7 left a comment

Choose a reason for hiding this comment

imbajin left a comment

Choose a reason for hiding this comment

Labels

4 participants

aroundabout commented Nov 6, 2023 •

edited

Loading

codecov bot commented Nov 6, 2023 •

edited

Loading

liuxiaocs7 commented Nov 10, 2023 •

edited

Loading

aroundabout commented Nov 10, 2023 •

edited

Loading