feature: optimize data receiving #181

coderzc · 2022-02-16T10:09:41Z

implement #165

use zero-copy optimize data recv
remove fail message
improve some config
support buffer file sort
support sort with buffer file and fix some unit test
create target file
rebuild sort module

javeme · 2022-02-17T05:31:09Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/network/TransportUtil.java

 buf.writeBytes(encoded);
 }
+
+ public static void setMaxBytesPreRead(Channel channel, int length) {


PreRead or PerRea?

javeme · 2022-02-17T05:31:53Z

...er-core/src/main/java/com/baidu/hugegraph/computer/core/network/buffer/FileRegionBuffer.java

+ this.length = length;
+ }
+
+ // Use zero-copy transform from socket channel to file


use "/**" for java doc

javeme · 2022-02-17T05:35:05Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/network/message/FailMessage.java

+ // Skip ackId
+ buf.skipBytes(Integer.BYTES);
+ // Skip partition
+ buf.skipBytes(Integer.BYTES);


skipBytes(Integer.BYTES + Integer.BYTES);

it seems more clear to skip them separately

javeme · 2022-02-17T05:35:50Z

...core/src/main/java/com/baidu/hugegraph/computer/core/network/netty/AbstractNettyHandler.java

 this.transportHandler().exceptionCaught(exception, connectionId);
 }

+ @Deprecated


won't fail any more?

at present the FailMessage seems there is useless but lack of increased the complexity of the implementation

javeme · 2022-02-17T05:38:55Z

...r-core/src/main/java/com/baidu/hugegraph/computer/core/network/netty/NettyServerHandler.java

- dataMessage.body());
-
- this.serverSession.onHandledData(requestId);
+ if (body instanceof FileRegionBuffer) {


can add two private methods for if-else branch?

javeme · 2022-02-17T05:43:35Z

...core/src/main/java/com/baidu/hugegraph/computer/core/network/netty/codec/MessageDecoder.java

+ if (this.fileRegionMode) {
+ return DataMessage.parseWithFileRegion(msgType, in);
+ } else {
+ return DataMessage.parseFrom(msgType, in);


also rename parseFrom to parseWithFileRegion style

javeme · 2022-02-17T05:49:29Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/receiver/MessageRecvManager.java

 }

+ @Override
+ public String genOutputPath(MessageType messageType, int partition) {


is it possible to keep an extension point that can use tmpfs?

javeme · 2022-02-17T06:04:12Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/sort/SorterImpl.java

+ Function<String, EntryIterator> fileToInput;
+ Function<String, KvEntryFileWriter> fileToWriter;
+
+ if (this.useZeroCopy) {


can use a set of subclasses to implement zero-copy mode to replace all the if (this.useZeroCopy)

can use a set of subclasses to implement zero-copy mode to replace all the if (this.useZeroCopy)
updated

javeme · 2022-02-17T06:05:04Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/network/netty/NettyProtocol.java

 * <pre>
 * +----------------------+
- * | LocalBuffer |
+ * | Local File  |


is there no local buffer mode any more?

javeme · 2022-02-17T07:34:01Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/sort/sorting/SortManager.java

+ if (this.threadNum(config) != 0) {
+ this.sortExecutor = ExecutorUtil.newFixedThreadPool(
+ this.threadNum(config), this.threadPrefix());
+ }


keep final mark and set sortExecutor=null

javeme · 2022-02-17T07:35:31Z

...uter-core/src/main/java/com/baidu/hugegraph/computer/core/receiver/MessageRecvPartition.java

+ Boolean zeroCopyMode =
+ config.get(ComputerOptions.TRANSPORT_ZERO_COPY_MODE);
+ if (!zeroCopyMode) {
+ this.recvBuffers = new MessageRecvBuffers(buffersLimit,


check other codes where used recvBuffers to avoid NPE

codecov · 2022-02-22T02:33:14Z

Codecov Report

Merging #181 (bc4e56e) into master (02a9361) will decrease coverage by 0.72%.
The diff coverage is 76.60%.

@@ Coverage Diff @@ ## master #181 +/- ## ============================================ - Coverage 87.33% 86.60% -0.73%  - Complexity 3150 3190 +40  ============================================ Files 332 339 +7 Lines 11806 12046 +240 Branches 1053 1070 +17 ============================================ + Hits 10311 10433 +122  - Misses 989 1095 +106  - Partials 506 518 +12

Impacted Files	Coverage Δ
...omputer/core/combiner/AbstractPointerCombiner.java	`100.00% <ø> (ø)`
...ph/computer/core/combiner/VertexValueCombiner.java	`78.26% <ø> (ø)`
...uter/core/common/exception/TransportException.java	`72.22% <ø> (-5.56%)`	⬇️
...ugegraph/computer/core/compute/ComputeManager.java	`86.84% <ø> (ø)`
...raph/computer/core/compute/FileGraphPartition.java	`86.97% <ø> (ø)`
...raph/computer/core/compute/input/MessageInput.java	`91.30% <ø> (ø)`
...h/computer/core/compute/input/ReusablePointer.java	`87.09% <ø> (ø)`
...om/baidu/hugegraph/computer/core/io/IOFactory.java	`70.58% <ø> (ø)`
...u/hugegraph/computer/core/io/StreamGraphInput.java	`97.43% <ø> (ø)`
.../hugegraph/computer/core/io/StreamGraphOutput.java	`96.96% <ø> (ø)`
... and 88 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 02a9361...bc4e56e. Read the comment docs.

javeme · 2022-03-02T07:45:01Z

There are still some comments need to be addressed

github-actions · 2022-04-01T21:11:48Z

Due to the lack of activity, the current pr is marked as stale and will be closed after 180 days, any update will remove the stale label

github-actions · 2022-05-02T21:07:34Z

Due to the lack of activity, the current pr is marked as stale and will be closed after 180 days, any update will remove the stale label

javeme · 2022-05-07T11:51:10Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/network/TransportUtil.java

 buf.writeBytes(encoded);
 }
+
+ public static void setMaxBytesPreRead(Channel channel, int length) {


PreRead or PerRead?

javeme · 2022-05-07T11:55:32Z

...uter-core/src/main/java/com/baidu/hugegraph/computer/core/receiver/MessageRecvPartition.java

+ waitSortTimeout);
+ this.sortBuffers = new MessageRecvBuffers(buffersLimit,
+ waitSortTimeout);
+ }


check other code used recvBuffers and sortBuffers, since recvBuffers/sortBuffers may be null

I checked there is no other reference when useFileRegion == true

javeme · 2022-05-12T06:47:57Z

can we try install hdfs through shell scrtipt:
https://github.com/beyondstorage/setup-hdfs/blob/master/src/setup-hdfs.ts#L13

coderzc · 2022-05-17T15:53:46Z

can we try install hdfs through shell scrtipt: https://github.com/beyondstorage/setup-hdfs/blob/master/src/setup-hdfs.ts#L13

I will fix it in #184

codecov · 2022-05-19T03:53:30Z

Codecov Report

Merging #181 (fd037a4) into master (bcaef85) will decrease coverage by 0.76%.
The diff coverage is 76.90%.

@@ Coverage Diff @@ ## master #181 +/- ## ============================================ - Coverage 86.79% 86.03% -0.77%  - Complexity 3169 3206 +37  ============================================ Files 334 341 +7 Lines 11962 12200 +238 Branches 1068 1085 +17 ============================================ + Hits 10383 10496 +113  - Misses 1067 1176 +109  - Partials 512 528 +16

Impacted Files	Coverage Δ
...omputer/core/combiner/AbstractPointerCombiner.java	`100.00% <ø> (ø)`
...ph/computer/core/combiner/VertexValueCombiner.java	`78.26% <ø> (ø)`
...uter/core/common/exception/TransportException.java	`72.22% <ø> (-5.56%)`	⬇️
...ugegraph/computer/core/compute/ComputeManager.java	`86.51% <ø> (ø)`
...raph/computer/core/compute/FileGraphPartition.java	`87.55% <ø> (ø)`
...raph/computer/core/compute/input/MessageInput.java	`91.30% <ø> (ø)`
...h/computer/core/compute/input/ReusablePointer.java	`87.09% <ø> (ø)`
...om/baidu/hugegraph/computer/core/io/IOFactory.java	`70.58% <ø> (ø)`
...u/hugegraph/computer/core/io/StreamGraphInput.java	`97.43% <ø> (ø)`
.../hugegraph/computer/core/io/StreamGraphOutput.java	`96.96% <ø> (ø)`
... and 89 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bcaef85...fd037a4. Read the comment docs.

corgiboygsj force-pushed the optimize_data_recv branch from 6fb7477 to 5709bc0 Compare February 16, 2022 10:10

coderzc force-pushed the optimize_data_recv branch 4 times, most recently from 865a71e to 24f806e Compare February 17, 2022 05:51

javeme reviewed Feb 17, 2022

View reviewed changes

coderzc force-pushed the optimize_data_recv branch 2 times, most recently from b77a046 to da0a0b9 Compare February 17, 2022 07:31

javeme reviewed Feb 17, 2022

View reviewed changes

coderzc force-pushed the optimize_data_recv branch 2 times, most recently from 7a483f1 to 690a2a5 Compare February 17, 2022 08:27

corgiboygsj force-pushed the optimize_data_recv branch 4 times, most recently from 62195d1 to fa07a59 Compare February 17, 2022 09:23

coderzc changed the title ~~Optimize data recv~~ optimize data receiving Feb 17, 2022

coderzc force-pushed the optimize_data_recv branch from c140833 to bc4e56e Compare February 25, 2022 05:58

coderzc marked this pull request as draft March 2, 2022 07:52

github-actions bot added the inactive label Apr 1, 2022

javeme removed the inactive label Apr 2, 2022

github-actions bot added the inactive label May 2, 2022

javeme removed the inactive label May 4, 2022

coderzc marked this pull request as ready for review May 7, 2022 09:24

javeme reviewed May 7, 2022

View reviewed changes

coderzc changed the title ~~optimize data receiving~~ feature: optimize data receiving May 10, 2022

javeme previously approved these changes May 10, 2022

View reviewed changes

coderzc dismissed javeme’s stale review via dfc33cb May 19, 2022 02:26

coderzc force-pushed the optimize_data_recv branch from 299582a to dfc33cb Compare May 19, 2022 02:26

javeme previously approved these changes May 19, 2022

View reviewed changes

coderzc and others added 15 commits May 19, 2022 14:27

use zero-copy optimize data recv

bf096d3

remove fail message

ee9c051

improve some config

81860a0

support buffer file sort

c9d76c5

imporve code

8c2e1a7

support sort with buffer file and fix some unit test

68dbc5c

create target file

4228f5e

fix test

d5f7736

improve config

87d2025

improve config

ed65be6

rebuild sort module

3a11686

fix unit test

96f63e2

improve some comment

564246b

fix function name (PerRead -> PerRead)

656c4bf

useFileBuffer -> useFileRegion

fd037a4

corgiboygsj dismissed javeme’s stale review via fd037a4 May 19, 2022 06:31

corgiboygsj force-pushed the optimize_data_recv branch from dfc33cb to fd037a4 Compare May 19, 2022 06:31

corgiboygsj approved these changes May 19, 2022

View reviewed changes

javeme approved these changes May 19, 2022

View reviewed changes

javeme merged commit 1800517 into master May 19, 2022

javeme deleted the optimize_data_recv branch May 19, 2022 09:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feature: optimize data receiving #181

feature: optimize data receiving #181

Uh oh!

coderzc commented Feb 16, 2022 •

edited

Loading

javeme Feb 17, 2022

javeme Feb 17, 2022

javeme Feb 17, 2022

coderzc May 7, 2022

javeme Feb 17, 2022

coderzc May 7, 2022

javeme Feb 17, 2022

javeme Feb 17, 2022

javeme Feb 17, 2022

javeme Feb 17, 2022

corgiboygsj Feb 17, 2022

javeme Feb 17, 2022

coderzc May 7, 2022

javeme Feb 17, 2022

coderzc May 7, 2022

javeme Feb 17, 2022

codecov bot commented Feb 22, 2022 •

edited

Loading

javeme commented Mar 2, 2022

github-actions bot commented Apr 1, 2022

github-actions bot commented May 2, 2022

javeme May 7, 2022

javeme May 7, 2022

coderzc May 10, 2022

javeme commented May 12, 2022

coderzc commented May 17, 2022

codecov bot commented May 19, 2022 •

edited

Loading

Labels

3 participants

feature: optimize data receiving #181

feature: optimize data receiving #181

Uh oh!

Conversation

coderzc commented Feb 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Feb 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

javeme commented Mar 2, 2022

github-actions bot commented Apr 1, 2022

github-actions bot commented May 2, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme commented May 12, 2022

coderzc commented May 17, 2022

codecov bot commented May 19, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Labels

3 participants

coderzc commented Feb 16, 2022 •

edited

Loading

codecov bot commented Feb 22, 2022 •

edited

Loading

codecov bot commented May 19, 2022 •

edited

Loading