Skip to content

Conversation

@vporyadke
Copy link
Collaborator

@vporyadke vporyadke commented Feb 7, 2025

Changelog entry

...

Changelog category

  • Documentation (changelog entry is not required)

Additional information

...

@vporyadke vporyadke requested a review from a team as a code owner February 7, 2025 16:02
@github-actions
Copy link

github-actions bot commented Feb 7, 2025

2025-02-07 16:03:29 UTC Pre-commit check linux-x86_64-relwithdebinfo for 9548108 has started.
2025-02-07 16:04:43 UTC Artifacts will be uploaded here
2025-02-07 16:07:12 UTC ya make is running...
🔴 2025-02-07 16:08:30 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Feb 7, 2025

✅ Documentation build

Revision built successfully
Revision preview link

@github-actions
Copy link

github-actions bot commented Feb 7, 2025

2025-02-07 16:09:51 UTC Pre-commit check linux-x86_64-release-asan for 9548108 has started.
2025-02-07 16:10:27 UTC Artifacts will be uploaded here
2025-02-07 16:13:24 UTC ya make is running...
🔴 2025-02-07 16:14:37 UTC Build failed, see the logs. Also see fail summary

@github-actions
Copy link

github-actions bot commented Feb 10, 2025

2025-02-10 11:46:24 UTC Pre-commit check linux-x86_64-relwithdebinfo for 2c4788c has started.
2025-02-10 11:46:37 UTC Artifacts will be uploaded here
2025-02-10 11:49:11 UTC ya make is running...
🟢 2025-02-10 11:50:47 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-10 11:50:54 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Feb 10, 2025

2025-02-10 11:46:27 UTC Pre-commit check linux-x86_64-release-asan for 2c4788c has started.
2025-02-10 11:46:46 UTC Artifacts will be uploaded here
2025-02-10 11:49:15 UTC ya make is running...
🟢 2025-02-10 11:50:44 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-10 11:50:50 UTC Build successful.

@github-actions
Copy link

✅ Documentation build

Revision built successfully
Revision preview link

@vporyadke vporyadke requested a review from CyberROFL February 12, 2025 11:08
@github-actions github-actions bot added documentation Improvements or additions to documentation and removed not-for-changelog labels Feb 12, 2025
@vporyadke vporyadke linked an issue Feb 12, 2025 that may be closed by this pull request
8 tasks
@vporyadke vporyadke requested a review from blinkov February 14, 2025 10:54
Co-authored-by: Ivan Blinkov <ivan@blinkov.ru>
@vporyadke vporyadke requested a review from blinkov February 19, 2025 10:00
@github-actions
Copy link

github-actions bot commented Feb 19, 2025

2025-02-19 10:02:49 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4413773 has started.
2025-02-19 10:03:07 UTC Artifacts will be uploaded here
2025-02-19 10:05:31 UTC ya make is running...
🟢 2025-02-19 10:05:38 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-19 10:05:43 UTC Build successful.

@github-actions
Copy link

✅ Documentation build

Revision built successfully
Revision preview link

@github-actions
Copy link

github-actions bot commented Feb 19, 2025

2025-02-19 10:04:23 UTC Pre-commit check linux-x86_64-release-asan for 4413773 has started.
2025-02-19 10:05:42 UTC Artifacts will be uploaded here
2025-02-19 10:07:58 UTC ya make is running...
🟢 2025-02-19 10:08:04 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-19 10:08:10 UTC Build successful.


To distribute tablets across nodes, Hive takes resource usage into account. For each tablet, the usage of four types of resources is tracked:

1. *CPU* — processor consumption, calculated as the number of microseconds spent on tablet work in the last second and converted to fractions of a core for visualization.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Converted to fractions of a core -- не понятно. Может сказать, что one second corresponds to 100% load of a single core?

3. *Network* — the amount of traffic generated by the tablet.
4. *Counter* — a fake resource used to implement even distribution of tablets. If a tablet has a nonzero consumption of any other resource, its Counter value is 0; otherwise, it is 1. This way, Counter is used for any tablets for which there is no data on real consumption, as well as for tablets where real consumption tracking is disabled. By default, this applies only to [columnar tables](../concepts/datamodel/table.md#column-oriented-tables).

Additionally, to determine overloaded nodes, metrics of resource usage for the node as a whole are used: memory consumption and processor resources in the actor system thread pools. These values are converted into a relative value (a number from 0 to 1), and their maximum is used as the node's overall resource consumption value—*Node usage*. Hive also applies aggregation over a window to all metrics to account for load spikes.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Additionally, to determine overloaded nodes, metrics of resource usage for the node as a whole are used: memory consumption and processor resources in the actor system thread pools. These values are converted into a relative value (a number from 0 to 1), and their maximum is used as the node's overall resource consumption value*Node usage*. Hive also applies aggregation over a window to all metrics to account for load spikes.
Additionally, to determine overloaded nodes, YDB uses memory consumption and processor resources in the actor system thread pools on each node. These values are converted into relative values (a number from 0 to 1). The sum of these relative values is used as the node's overall resource consumption value*Node usage*. Hive also applies aggregation over a window to all metrics to account for load spikes.

The creation and deletion of tablets is initiated by the [SchemeShard](../concepts/glossary.md#scheme-shard) tablet. When a tablet is created, Hive assigns it a unique TabletId, fills in [TabletStorageInfo](general-schema.md#history), selects the most suitable node, and sends a command to start the tablet on that node. In some abnormal situations, a tablet may interrupt its operation, in which case the node on which it was running sends a message to Hive. Hive also assumes that if the connection with a certain node is lost, the tablets running on it have stopped. In such cases, Hive restarts the tablets on other nodes, increasing the generation.

In a {{ ydb-short-name }} cluster, there is a root Hive responsible for the [system tablets](../concepts/glossary.md#tablet-types) of all databases in the cluster. The Hive of a specific database, in turn, is responsible for the tablets servicing the user load of that database. All nodes in the cluster are registered in the root Hive, while only the compute nodes of a specific database are registered in that database's Hive. When a node registers, it informs Hive of the types of tablets and the number of tablets that can be run on it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Попробовал разбить на 2 пункта:

A {{ ydb-short-name }} cluster runs multiple Hives:

  • A single root Hive is responsible for the system tablets of all databases in the cluster. All nodes in the cluster are registered in the root Hive.
  • A database Hive (one per database) is responsible for the tablets servicing the user load of a specific database. Only the compute nodes of a specific database are registered in that database Hive.

When a new node registers, it informs [какой именно??] Hive of the types of tablets and the number of tablets that can be run on it.

1. *CPU* — processor consumption, calculated as the number of microseconds spent on tablet work in the last second and converted to fractions of a core for visualization.
2. *Memory* — the amount of RAM consumed by the tablet.
3. *Network* — the amount of traffic generated by the tablet.
4. *Counter* — a fake resource used to implement even distribution of tablets. If a tablet has a nonzero consumption of any other resource, its Counter value is 0; otherwise, it is 1. This way, Counter is used for any tablets for which there is no data on real consumption, as well as for tablets where real consumption tracking is disabled. By default, this applies only to [columnar tables](../concepts/datamodel/table.md#column-oriented-tables).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
4. *Counter* — a fake resource used to implement even distribution of tablets. If a tablet has a nonzero consumption of any other resource, its Counter value is 0; otherwise, it is 1. This way, Counter is used for any tablets for which there is no data on real consumption, as well as for tablets where real consumption tracking is disabled. By default, this applies only to [columnar tables](../concepts/datamodel/table.md#column-oriented-tables).
4. *Counter* — a fake resource used to implement even distribution of tablets. If a tablet has a nonzero consumption of any other resource, its Counter value is 0; otherwise, it is 1. This way, Counter is used for any tablets for which there is no data on CPU, memory, and network consumption. This resource type is also used for tablets, in which resource consumption tracking is disabled. By default, this applies only to [column-oriented tables](../concepts/datamodel/table.md#column-oriented-tables).
Co-authored-by: anton-bobkov <anton-bobkov@ydb.tech>
@github-actions
Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 16:04:36 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1ffbd9a has started.
2025-02-20 16:04:51 UTC Artifacts will be uploaded here
2025-02-20 16:07:14 UTC ya make is running...
🟢 2025-02-20 16:07:20 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-20 16:07:27 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 16:07:02 UTC Pre-commit check linux-x86_64-release-asan for 1ffbd9a has started.
2025-02-20 16:08:10 UTC Artifacts will be uploaded here
2025-02-20 16:10:34 UTC ya make is running...
🟢 2025-02-20 16:10:40 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-20 16:10:46 UTC Build successful.

@github-actions
Copy link

✅ Documentation build

Revision built successfully
Revision preview link

Co-authored-by: anton-bobkov <anton-bobkov@ydb.tech>
@github-actions
Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 22:12:12 UTC Pre-commit check linux-x86_64-release-asan for fee1f68 has started.
2025-02-20 22:12:24 UTC Artifacts will be uploaded here
2025-02-20 22:14:51 UTC ya make is running...
🟢 2025-02-20 22:14:57 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-20 22:15:03 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 22:12:27 UTC Pre-commit check linux-x86_64-relwithdebinfo for fee1f68 has started.
2025-02-20 22:12:40 UTC Artifacts will be uploaded here
2025-02-20 22:14:58 UTC ya make is running...
🟢 2025-02-20 22:15:04 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-20 22:15:10 UTC Build successful.

@github-actions
Copy link

✅ Documentation build

Revision built successfully
Revision preview link

@github-actions
Copy link

❌ Documentation build

Revision build failed

Build logs

Errors (1)

❌ /en/contributor/hive.md: 14: MD032/blanks-around-lists Lists should be surrounded by blank lines [Context: "* A single root Hive respons..."]

@github-actions
Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 22:25:01 UTC Pre-commit check linux-x86_64-release-asan for ad09048 has started.
2025-02-20 22:25:34 UTC Artifacts will be uploaded here
2025-02-20 22:28:33 UTC ya make is running...
🟢 2025-02-20 22:28:38 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-20 22:28:44 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Feb 20, 2025

2025-02-20 22:25:44 UTC Pre-commit check linux-x86_64-relwithdebinfo for ad09048 has started.
2025-02-20 22:25:55 UTC Artifacts will be uploaded here
2025-02-20 22:28:12 UTC ya make is running...
🟢 2025-02-20 22:28:17 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-20 22:28:23 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Feb 21, 2025

2025-02-21 01:13:46 UTC Pre-commit check linux-x86_64-relwithdebinfo for eb2445b has started.
2025-02-21 01:13:55 UTC Artifacts will be uploaded here
2025-02-21 01:16:13 UTC ya make is running...
🟢 2025-02-21 01:16:19 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-21 01:16:26 UTC Build successful.

@github-actions
Copy link

github-actions bot commented Feb 21, 2025

2025-02-21 01:16:54 UTC Pre-commit check linux-x86_64-release-asan for eb2445b has started.
2025-02-21 01:17:05 UTC Artifacts will be uploaded here
2025-02-21 01:19:26 UTC ya make is running...
🟢 2025-02-21 01:19:32 UTC Tests successful.

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
0 0 0 0 0 0

🟢 2025-02-21 01:19:37 UTC Build successful.

@github-actions
Copy link

✅ Documentation build

Revision built successfully
Revision preview link

@blinkov blinkov enabled auto-merge (squash) February 21, 2025 01:20
@blinkov blinkov merged commit 8535304 into ydb-platform:main Feb 21, 2025
13 checks passed
lberserq pushed a commit to lberserq/ydb that referenced this pull request Mar 3, 2025
Co-authored-by: Ivan Blinkov <ivan@ydb.tech> Co-authored-by: anton-bobkov <anton-bobkov@ydb.tech>
blinkov added a commit that referenced this pull request Mar 21, 2025
Co-authored-by: Ivan Blinkov <ivan@ydb.tech> Co-authored-by: anton-bobkov <anton-bobkov@ydb.tech>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

4 participants