feat: Dynamic flow control p2 #670
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
google-cla bot added the cla: yes 
product-auto-label bot added the api: bigtable 
Codecov Report
@@ Coverage Diff @@ ## dynamic_flow_control #670 +/- ## ======================================================= Coverage ? 59.20% Complexity ? 19 ======================================================= Files ? 2 Lines ? 125 Branches ? 18 ======================================================= Hits ? 74 Misses ? 34 Partials ? 17 Continue to review full report at Codecov.
|
mutianf force-pushed the dynamic_flow_control_p2 branch from ffd9de9 to 657c652 Compare mutianf force-pushed the dynamic_flow_control_p2 branch from 657c652 to 7a4ff43 Compare 
mutianf commented Mar 10, 2021
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java Show resolved Hide resolved
igorbernstein2 requested changes Mar 24, 2021
| flowControlEvents.getLastFlowControlEvent() == null | ||
| ? false | ||
| : (now - flowControlEvents.getLastFlowControlEvent().getTimestampMs() | ||
| <= TimeUnit.MINUTES.toMillis(5)); |
Contributor
There was a problem hiding this comment.
Can you extract the thresholds to named constants?
ie. MIN_THROTTLE_INTERVAL = Duration.ofMinutes(5)
CONCURRENCY_DECREASE_STEP = 0.3;
CONCURRENCY_INCREASE_STEP = ...
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java Outdated Show resolved Hide resolved
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java Outdated Show resolved Hide resolved
Comment on lines 34 to 35
| private final FlowControlEventStats flowControlEvents; | ||
| private final DynamicFlowControlStats dynamicFlowControlStats; |
Contributor
There was a problem hiding this comment.
I think we can improve on class naming here. The names are too similar which makes hard to remember what each does. Unfortunately I dont have good suggestions :(
Contributor Author
There was a problem hiding this comment.
Yeah, I noticed this too but also don't have any better idea 😢
But after flowControlEvents is moved to flowController I think it's less confusing now?
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java Outdated Show resolved Hide resolved
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java Show resolved Hide resolved
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java Outdated Show resolved Hide resolved
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java Outdated Show resolved Hide resolved
...loud-bigtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/EnhancedBigtableStub.java Outdated Show resolved Hide resolved
| | ||
| UnaryCallable<MutateRowsRequest, Void> flowControlCallable = null; | ||
| if (settings.bulkMutateRowsSettings().isLatencyBasedThrottlingEnabled()) { | ||
| long flowControlAdjustingIntervalMs = TimeUnit.SECONDS.toMillis(20); |
Contributor
There was a problem hiding this comment.
this should be a constant somewhere
igorbernstein2 approved these changes Mar 29, 2021
Contributor
igorbernstein2 left a comment
igorbernstein2 left a comment There was a problem hiding this comment.
lgtm with a minor nit
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java Outdated Show resolved Hide resolved
...igtable/src/main/java/com/google/cloud/bigtable/data/v2/stub/DynamicFlowControlCallable.java Outdated Show resolved Hide resolved
mutianf added a commit to mutianf/java-bigtable that referenced this pull request Apr 10, 2021
* feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant
mutianf added a commit to mutianf/java-bigtable that referenced this pull request Apr 10, 2021
* feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant
mutianf added a commit to mutianf/java-bigtable that referenced this pull request Apr 10, 2021
* feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant
mutianf added a commit to mutianf/java-bigtable that referenced this pull request Apr 10, 2021
* feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant
mutianf added a commit to mutianf/java-bigtable that referenced this pull request Apr 12, 2021
* feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant
igorbernstein2 pushed a commit that referenced this pull request Apr 13, 2021
* feat: dynamic flow control part 1 (#620) * feat: dynamic flow control p1, add settings * update documents * move flow controller instantiation to stub.create * fix test * add some tests * split out settings to BatchCallSettings * add getters for target rpc latency * fix doc * more changes in doc * update document and checks on parameters * add more docs on flow control configs * fix test * split up enable/disable settings * feat: Dynamic flow control p2 (#670) * feat: dynamic flow control for bulk mutation batcher * improve documents * add integration test * clean up tests and formatting * add more comments * moved FlowControlEventStats into FlowController * updates based on review p1 * updates based on review p2 * updates on review * update constant * fix tests * make row key prefix random * updates based on review
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Suggestion cannot be applied right now. Please check back later.
Part 2 of go/veneer-dynamic-flow-control.
Adding dynamic flow control callable to bulk mutation
Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:
Fixes #<issue_number_goes_here> ☕️