add BufferedFileDataOutput and BufferedFileDataInput #15

houzhizhen · 2021-02-25T06:55:34Z

No description provided.

codecov-io · 2021-02-25T06:57:40Z

Codecov Report

Merging #15 (2104b2c) into master (bb462d3) will increase coverage by 2.47%.
The diff coverage is 94.39%.

@@ Coverage Diff @@ ## master #15 +/- ## ============================================ + Coverage 77.83% 80.31% +2.47%  - Complexity 631 746 +115  ============================================ Files 68 72 +4 Lines 2148 2474 +326 Branches 176 216 +40 ============================================ + Hits 1672 1987 +315  - Misses 370 371 +1  - Partials 106 116 +10

Impacted Files	Coverage Δ	Complexity Δ
...aidu/hugegraph/computer/core/common/Constants.java	`50.00% <ø> (ø)`	`1.00 <0.00> (ø)`
...hugegraph/computer/core/io/GraphOutputFactory.java	`33.33% <ø> (ø)`	`2.00 <0.00> (ø)`
...ph/computer/core/io/OptimizedStreamGraphInput.java	`100.00% <ø> (ø)`	`3.00 <0.00> (ø)`
...h/computer/core/io/OptimizedStreamGraphOutput.java	`100.00% <ø> (ø)`	`3.00 <0.00> (ø)`
...egraph/computer/core/io/UnsafeByteArrayOutput.java	`97.05% <85.71%> (+6.76%)`	`33.00 <2.00> (+3.00)`
...gegraph/computer/core/io/BufferedStreamOutput.java	`89.18% <89.18%> (ø)`	`24.00 <24.00> (?)`
.../hugegraph/computer/core/io/BufferedFileInput.java	`93.65% <93.65%> (ø)`	`20.00 <20.00> (?)`
...hugegraph/computer/core/io/BufferedFileOutput.java	`93.75% <93.75%> (ø)`	`22.00 <22.00> (?)`
...gegraph/computer/core/io/UnsafeByteArrayInput.java	`97.05% <95.45%> (-0.51%)`	`34.00 <10.00> (+8.00)`	⬇️
...ugegraph/computer/core/io/BufferedStreamInput.java	`98.71% <98.71%> (ø)`	`25.00 <25.00> (?)`
... and 7 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb462d3...2104b2c. Read the comment docs.

javeme · 2021-02-26T05:19:28Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileDataInput.java

+ int readLen = (int) Math.min(bufferSize, this.fileLength);
+ this.file.readFully(this.buffer(), 0, readLen);
+ this.fileOffset = readLen;
+ super.limit(readLen);


call fillBuffer

javeme · 2021-02-26T05:27:20Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/UnsafeByteArrayInput.java

+ int remaining = this.remaining();
+ if (remaining > 0) {
+ System.arraycopy(this.buffer, this.position,
+ this.buffer,0, remaining);


add a space before 0

javeme · 2021-02-26T05:30:21Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/UnsafeByteArrayInput.java

+ * Cut the content from 0 to position and copy the content from position
+ * to the end to 0.
+ */
+ protected void cutReadBuffer() {


shiftBuffer() is ok

javeme · 2021-02-26T05:33:50Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileDataOutput.java

+ super.write(b, off, len);
+ return;
+ }
+ this.writeBuffer();


rename to flushBuffer()

javeme · 2021-02-26T05:44:57Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileDataInput.java

+ int remaining = super.remaining();
+ super.readFully(b, off, remaining);
+ this.file.readFully(b, off + remaining, len - remaining);
+ this.fileOffset += len;


add a method fillBuffer()

part of the data needed is in buffer, the other part is in file, and the length of data is beyond the buffer size, so read from two places.

we can add length parameter to fillBuffer()

javeme · 2021-02-26T05:50:36Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileDataInput.java

+import com.baidu.hugegraph.util.E;
+
+public class BufferedFileDataInput extends UnsafeByteArrayInput
+ implements RandomAccessInput {


don't implement RandomAccessInput, since UnsafeByteArrayInput implements RandomAccessInput

javeme · 2021-02-26T05:58:04Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileDataInput.java

+ E.checkArgument(bufferSize >= 8,
+ "The parameter bufferSize must be >= 8");
+ this.file = file;
+ this.fileLength = file.length();


remove this field

fileLength is used several time, we call file.length() every time?

seems the fileLength is not used everywhere after added fillBuffer() method

it is used in seek(long position). The position may be far from the current position

javeme · 2021-02-26T06:06:55Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileDataInput.java

+ private final RandomAccessFile file;
+ private final long fileLength;
+
+ public BufferedFileDataInput(File file) throws IOException {


add RandomAccessInputStream class to support InputStream as parameter

javeme · 2021-02-26T06:07:31Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileDataInput.java

+import com.baidu.hugegraph.computer.core.common.exception.ComputerException;
+import com.baidu.hugegraph.util.E;
+
+public class BufferedFileDataInput extends UnsafeByteArrayInput


rename to BufferedFileInput

javeme · 2021-02-27T12:57:31Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileDataInput.java

+ int remaining = super.remaining();
+ super.readFully(b, off, remaining);
+ this.file.readFully(b, off + remaining, len - remaining);
+ this.fileOffset += len;


we can add length parameter to fillBuffer()

javeme · 2021-02-27T12:57:36Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileDataInput.java

+ super.seek(0L);
+ this.file.readFully(this.buffer(), 0, readLen);
+ super.limit(readLen);
+ this.fileOffset = position + readLen;


also call fillBuffer()

javeme · 2021-02-27T12:59:10Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileDataInput.java

+ }
+
+ @Override
+ public void readFully(byte[] b, int off, int len) throws IOException {


should also deal with buffer overflow for other read* methods like readLong?

no need, since the buffer size must be >=8, and if there is no enough data in the buffer, the overwritten require method will fillBuffer again.

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileOutput.java

javeme · 2021-03-03T09:24:33Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileOutput.java

- long bufferAvailable = this.bufferSize - position;
- if (bufferAvailable >= size) {
- super.require(size);
+ long available = this.bufferSize - position;


long available = this.bufferSize - super.position();

javeme · 2021-03-03T09:24:48Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedInputStream.java

+ }
+
+ public BufferedInputStream(InputStream in, int bufferSize)
+ throws IOException {


javeme · 2021-03-03T09:26:42Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedInputStream.java

+
+public class BufferedInputStream extends UnsafeByteArrayInput {
+
+ private long inOffset;


inputOffset

javeme · 2021-03-03T09:27:01Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedInputStream.java

+ if (this.remaining() >= size) {
+ return;
+ }
+ shiftAndFillBuffer();


javeme · 2021-03-03T09:34:19Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+ }
+
+ @Test
+ public void testInt() throws IOException {


testWriteInt

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

javeme · 2021-03-03T09:36:07Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+ output.writeInt(Integer.MAX_VALUE);
+ output.writeInt(Integer.MIN_VALUE);
+ }
+ LOG.info("file.length:{}", file.length());


javeme · 2021-03-03T09:36:34Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+ }
+ }
+
+ private File createTempFile() throws IOException {


javeme · 2021-03-03T11:50:47Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedOutputStream.java

- (this.osOffset + this.bufferSize) + ")");
+ "of range [" + this.outputOffset + ", " +
+ (this.outputOffset + this.bufferSize) +
+ ")");


prefer to use String.format()

javeme · 2021-03-03T11:51:33Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileTest.java

 }

- private BufferedFileOutput createOutput(File file)
+ private static BufferedFileOutput createOutput(File file)


computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileTest.java

javeme · 2021-03-03T11:52:04Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

 Assert.assertThrows(IOException.class, () -> {
 input.seek(size * 4 + 1);
+ }, e -> {
+ Assert.assertContains("Can't seek at position ",


remove space

javeme · 2021-03-03T11:52:10Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

 Assert.assertThrows(IOException.class, () -> {
 output.seek(output.position() - BUFFER_SIZE - 2);
+ }, e -> {
+ Assert.assertContains("out of range ", e.getMessage());


javeme · 2021-03-03T12:28:19Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+ }
+
+ @Test
+ public void testLargeByteArray() throws IOException {


javeme · 2021-03-03T12:28:47Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+
+ byte[] arrayRead = new byte[arraySize];
+ try (DataInputStream dis = new DataInputStream(
+ new FileInputStream(file))) {


prefer to align with new DataInputStream

javeme · 2021-03-03T12:31:02Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+ output.writeInt(i);
+ }
+ output.writeInt(200, 1);
+ // Previous buffer


test 5 kind of position: buffer_start ~ buffer_end

The five kind of position include start point of current buffer, middle point of current buffer, end point of current buffer, before the current buffer, and current buffer, and the last two kind of position throws exception.

javeme · 2021-03-03T12:32:10Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+ for (int i = 0; i < size; i++) {
+ output.seek(i * 4);
+ output.writeInt(i);
+


javeme · 2021-03-03T12:32:38Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+ }
+
+ @Test
+ public void testSeekMoreThanBuffer() throws IOException {


overflow and underflow

This test case is deleted, and overflow and underflow test is in testSeekOutRange

javeme

please addressed all the comments

javeme · 2021-03-07T02:51:40Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedInputStream.java

- throw new IOException("The position is beyond the buffer");
+ throw new IOException(String.format(
+ "The seek position %s is before the start position of " +
+ "buffer %s",


The seek position %s underflows the start position %s of the buffer

javeme · 2021-03-07T02:58:56Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedInputStream.java

+ /*
+ * The reason to seek to the position beyond the current buffer is
+ * the user may need to skip the data unread and known the position of
+ * the data needed.


The reason for seeking beyond the current buffer location is that the user may need to skip unread data and know the offset of the required data.

javeme · 2021-03-07T03:05:22Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

 output.seek((size - 2) * 4);
 output.writeInt(Integer.MAX_VALUE);
 output.writeInt(Integer.MIN_VALUE);
+ // The position is before the buffer


update all the "before" to "underflow"

javeme · 2021-03-07T03:06:18Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/common/Constants.java

 public static final int UINT16_MAX = 0xffff;
 public static final long UINT32_MAX = 0xffffffffL;
+
+ public static final int DEFAULT_BUFFER_SIZE = 8192;


not addressed comments: Bytes.KB * 8

javeme · 2021-03-08T05:35:28Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamInput.java

- "The seek position %s is before the start position of " +
- "buffer %s",
- position, this.inputOffset - super.limit()));
+ "The seek position %s is underflow the start position " +


is underflow => underflows

javeme · 2021-03-08T05:35:48Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamInput.java

- public long skip(long n) throws IOException {
- E.checkArgument(n >= 0, "The parameter n must be >=0, but got %s", n);
+ public long skip(long bytesToSkip) throws IOException {
+ E.checkArgument(bytesToSkip >= 0,


javeme · 2021-03-08T05:36:43Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamOutput.java

- "The parameter bytesToSkip must be <= " +
- "Integer.MAX_VALUE");
+ public long skip(long bytesToSkip) throws IOException {
+ E.checkArgument(bytesToSkip >= 0,


javeme · 2021-03-08T05:37:58Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileTest.java

 output.writeByte(1);
+ Assert.assertThrows(IllegalArgumentException.class, () -> {
+ output.skip(-1);
+ });


assert message

javeme · 2021-03-08T05:38:02Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileTest.java

 Assert.assertEquals((byte) 1, input.readByte());
+ Assert.assertThrows(IllegalArgumentException.class, () -> {
+ input.skip(-1);
+ });


assert message

javeme · 2021-03-08T05:40:00Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileInput.java

 private void fillBuffer() throws IOException {
 long fileLength = this.file.length();
- int readLen = Math.min(this.bufferSize - super.limit(),
+ int readLen = Math.min(this.bufferSize - this.limit(),


use this.remaining() instead

read fill the buffer after the limit, remaining is the buffer before the limit.
public int remaining() {
return this.limit - this.position;
}

javeme · 2021-03-08T05:45:22Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileInput.java

 if (position < this.fileOffset &&
 position >= this.fileOffset - this.bufferSize) {
- super.seek(this.bufferSize - (this.fileOffset - position));
+ super.seek(this.limit() - (this.fileOffset - position));


this.limit() and this.bufferSize are the same value?

Not the same value. The last buffer may not equals bufferSize.

bufferStart = this.fileOffset - this.limit
put position >= bufferStart before position < this.fileOffset
super.seek(position - bufferStart)

javeme · 2021-03-08T06:50:45Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileTest.java

 output.skip(-1);
+ }, e -> {
+ e.getMessage().contains("The parameter bytesToSkip must " +
+ "be >=0");


expect space ">=0"

javeme · 2021-03-08T06:53:46Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileInput.java

 if (position < this.fileOffset &&
 position >= this.fileOffset - this.bufferSize) {
- super.seek(this.bufferSize - (this.fileOffset - position));
+ super.seek(this.limit() - (this.fileOffset - position));


javeme · 2021-03-08T06:54:58Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamInput.java

+ int expectLen = Math.min(skipLen, this.bufferSize);
+ int readLen = this.input.read(buffer, 0, expectLen);
+ if (readLen == -1) {
+ throw new IOException("Reach the end of input stream");


improve message

javeme · 2021-03-08T06:57:46Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamInput.java

+ }
+ this.shiftAndFillBuffer();
+ if (size > this.limit()) {
+ throw new IOException("Can't read " + size + " bytes");


improve message, like "read %size bytes from position overflows buffer %this.limit()"

javeme · 2021-03-08T07:01:39Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/UnsafeByteArrayInput.java

- private void require(int size) {
+ @Override
+ public void close() throws IOException {
+ // Do nothing


javeme · 2021-03-08T07:02:31Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/UnsafeByteArrayInput.java

+
+ protected void limit(int limit) {
+ E.checkArgument(limit <= this.buffer.length,
+ "The limit must be >= buffer.length: {}",


buffer length %s

javeme · 2021-03-08T07:03:33Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+ Assert.assertContains("is out of range",
+ e.getMessage());
+ });
+ Assert.assertThrows(IOException.class, () -> {


Behind buffer

overflows the buffer is better.

javeme · 2021-03-08T07:03:53Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+
+ byte[] arrayRead = new byte[10];
+ try (DataInputStream dis = new DataInputStream(
+ new FileInputStream(file))) {


javeme · 2021-03-09T03:49:17Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/UnsafeByteArrayInput.java

 @Override
 public void close() throws IOException {
- // Do nothing
+ // Pass


lower case pass

javeme · 2021-03-09T03:50:20Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/UnsafeByteArrayOutput.java

 @Override
 public void close() throws IOException {
- // Do nothing
+ // Pass


javeme · 2021-03-09T03:50:48Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamInput.java

 if (size > this.limit()) {
- throw new IOException("Can't read " + size + " bytes");
+ throw new IOException(String.format(
+ "Read %s bytes from position overflows buffer %s",


position %s

javeme · 2021-03-09T03:51:05Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamOutput.java

- "The position %s is out of range [%s, %s]",
- position, this.outputOffset,
- this.position() - 4));
+ "Write int from position %s overflows the write " +


Write int to position

javeme · 2021-03-09T08:15:26Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamOutput.java

 } else {
 throw new IOException(String.format(
- "Write int from position %s overflows the write " +
+ "Write int to position %s overflows the write " +


this.position() - 4
use INT_SIZE

javeme · 2021-03-09T08:16:25Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileInput.java

+ private long fileOffset;
+
+ public BufferedFileInput(File file) throws IOException {
+ this(new RandomAccessFile(file, "r"), Constants.DEFAULT_BUFFER_SIZE);


const var "r"

javeme · 2021-03-09T08:21:26Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileInput.java

 if (position < this.fileOffset &&
 position >= this.fileOffset - this.bufferSize) {
- super.seek(this.bufferSize - (this.fileOffset - position));
+ super.seek(this.limit() - (this.fileOffset - position));


bufferStart = this.fileOffset - this.limit
put position >= bufferStart before position < this.fileOffset
super.seek(position - bufferStart)

javeme · 2021-03-09T08:21:55Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileInput.java

+ @Override
+ public void seek(long position) throws IOException {
+ if (position < this.fileOffset &&
+ position >= this.fileOffset - this.bufferSize) {


-this.limit

javeme · 2021-03-09T08:25:25Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileInput.java

+ if (this.bufferSize >= size) {
+ this.shiftAndFillBuffer();
+ } else {
+ throw new ComputerException("Should not reach here");


comment: Should not reach here
throw XxException

javeme · 2021-03-09T08:25:35Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileInput.java

+ this.fileOffset = position;
+ this.fillBuffer();
+ }
+


javeme · 2021-03-09T08:27:40Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileOutput.java

+ return;
+ }
+ this.flushBuffer();
+ assert size <= this.bufferAvailable();


javeme · 2021-03-09T08:28:36Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileOutput.java

+ private long fileOffset;
+
+ public BufferedFileOutput(File file) throws FileNotFoundException {
+ this(new RandomAccessFile(file, "rw"), Constants.DEFAULT_BUFFER_SIZE);


javeme · 2021-03-09T08:29:43Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileOutput.java

+ return;
+ }
+ this.flushBuffer();
+ if (len <= this.bufferSize) {


this.bufferCapacity

javeme · 2021-03-09T08:30:39Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileInput.java

+ E.checkArgument(bufferSize >= 8,
+ "The parameter bufferSize must be >= 8");
+ this.file = file;
+ this.bufferSize = bufferSize;


bufferCapacity

Linary · 2021-03-10T09:00:05Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileInput.java

+ /*
+ * The buffer capacity must be >= 8, read primitive data like int,
+ * long, float, double can be read from buffer. Only read bytes may
+ * exceed the limit, and read bytes using readFully is overrode


Linary · 2021-03-10T09:01:06Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileOutput.java

+ * The buffer capacity must be >= 8, write primitive data like int,
+ * long, float, double can be write to buffer after flush buffer.
+ * Only write bytes may exceed the limit, and write bytes using
+ * write(byte[] b) is overrode in this class. In conclusion, the


Linary · 2021-03-10T09:04:26Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileInput.java

+ this.file.close();
+ }
+
+ protected void require(int size) throws IOException {


add @OverRide

Linary · 2021-03-10T10:00:21Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/CsvStructGraphOutputTest.java

- try (FileOutputStream fos = new FileOutputStream(file);
-  DataOutputStream dos = new DataOutputStream(fos)) {
+ try {
+ BufferedFileOutput dos = new BufferedFileOutput(file);


try (BufferedFileOutput dos = new BufferedFileOutput(file)) {
...
}

Can't read the data in try, the dos in try only auto close after try.

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/CsvStructGraphOutputTest.java

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/JsonStructGraphOutputTest.java

Linary · 2021-03-10T11:07:12Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamOutput.java

+ throw new IOException(String.format(
+ "The position %s is out of range [%s, %s]",
+ position, this.outputOffset,
+ this.outputOffset + this.bufferCapacity));


upper bound is this.position()

Linary · 2021-03-10T11:16:58Z

computer-core/src/main/java/com/baidu/hugegraph/computer/core/io/StreamGraphInput.java

 return this.in.readInt();
 }

+ public int readFullInt() throws IOException {


javeme · 2021-03-10T12:09:27Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileTest.java

+ }
+ endTime = System.currentTimeMillis();
+ time = endTime - startTime;
+ LOG.info("Read {} bytes use DataInputStream.readFully takes{} ms",


expect space after "takes"

javeme · 2021-03-10T12:09:51Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileTest.java

+
+ private static BufferedFileOutput createOutput(File file)
+ throws FileNotFoundException {
+ return new BufferedFileOutput(new RandomAccessFile(file, "rw"),


javeme · 2021-03-10T12:09:57Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedFileTest.java

+
+ private static BufferedFileInput createInput(File file)
+ throws IOException {
+ return new BufferedFileInput(new RandomAccessFile(file, "rw"),


javeme · 2021-03-10T12:11:10Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+ // Middle of current buffer
+ output.writeInt(200, 2);
+ output.writeInt(252, 3);
+ // Previous buffer


update comment

javeme · 2021-03-10T12:11:18Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+ Assert.assertContains("underflows the start position",
+ e.getMessage());
+ });
+ Assert.assertThrows(IOException.class, () -> {


add comment

javeme · 2021-03-10T12:21:48Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+ Assert.assertContains("out of range", e.getMessage());
+ });
+
+


empty lines

javeme · 2021-03-10T12:24:45Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/CsvStructGraphOutputTest.java

 GraphOutputFactory.create(
 OutputFormat.CSV, dos);
 output.writeVertex(vertex);
+ dos.close();


prefer to call flush()

close is better, it calls flush and close other resource.

javeme · 2021-03-10T12:32:01Z

computer-test/src/main/java/com/baidu/hugegraph/computer/core/io/BufferedStreamTest.java

+
+ @Test
+ public void testConstructor() throws IOException {
+ File file = this.createTempFile();


javeme requested changes Feb 26, 2021

View reviewed changes

javeme reviewed Feb 26, 2021

View reviewed changes

javeme reviewed Feb 27, 2021

View reviewed changes

javeme reviewed Mar 3, 2021

View reviewed changes

javeme requested changes Mar 3, 2021

View reviewed changes

javeme requested changes Mar 7, 2021

View reviewed changes

houzhizhen force-pushed the input-output branch from 50a2ec3 to db11851 Compare March 8, 2021 03:28

javeme reviewed Mar 8, 2021

View reviewed changes

javeme reviewed Mar 9, 2021

View reviewed changes

javeme previously approved these changes Mar 10, 2021

View reviewed changes

coderzc previously approved these changes Mar 10, 2021

View reviewed changes

Linary requested changes Mar 10, 2021

View reviewed changes

houzhizhen dismissed stale reviews from coderzc and javeme via 20b4c14 March 10, 2021 11:44

Linary previously approved these changes Mar 10, 2021

View reviewed changes

javeme previously approved these changes Mar 10, 2021

View reviewed changes

javeme reviewed Mar 10, 2021

View reviewed changes

houzhizhen added 10 commits March 11, 2021 11:07

add BufferedFileDataOutput and BufferedFileDataInput

d871407

tiny improve

d2712d4

add BufferedInputStream and BufferedOutputStream

9cdfe4a

remove fileLength and adjust the fields order

a037fc7

tiny improve

ea1d279

tiny improve

b2593ff

tiny improve

d3cac4a

tiny improve

6046fca

tiny improve

6cf919e

tiny improve

450ca74

houzhizhen added 5 commits March 11, 2021 11:07

tiny improve

75b3663

tiny improve

e8a2b2a

change bufferSize to bufferCapacity

6b2e721

add Override annotation

52b4ff2

update "rw","r" in test case with const

2104b2c

houzhizhen dismissed stale reviews from javeme and Linary via 2104b2c March 11, 2021 03:08

houzhizhen force-pushed the input-output branch from f4e4d32 to 2104b2c Compare March 11, 2021 03:08

javeme approved these changes Mar 11, 2021

View reviewed changes

Linary approved these changes Mar 11, 2021

View reviewed changes

houzhizhen merged commit 128ce85 into master Mar 11, 2021

houzhizhen deleted the input-output branch March 11, 2021 03:41

yandababa2 pushed a commit to yandababa2/hugegraph-computer that referenced this pull request Oct 22, 2021

add HugeOutput for DegreeCentrality (apache#15)

1c5f7e2


		public class BufferedInputStream extends UnsafeByteArrayInput {

		private long inOffset;

add BufferedFileDataOutput and BufferedFileDataInput #15

add BufferedFileDataOutput and BufferedFileDataInput #15

Uh oh!

Conversation

houzhizhen commented Feb 25, 2021

codecov-io commented Feb 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme Feb 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

javeme left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Feb 25, 2021 •

edited

Loading

javeme Feb 26, 2021 •

edited

Loading