Try to preserve in-memory state (locks in particular) across restarts #12567

snaury · 2024-12-12T13:24:02Z

Changelog entry

Try to preserve in-memory state (locks in particular) across datashard restarts.

Changelog category

Improvement

Additional information

We now migrate in-memory state and in-memory locks in particular, which should reduce ABORTED errors due to lost locks while rebalancing tablets between nodes, and might improve shard startup time due to less waiting for mediator state restore.

Related to #11561.

github-actions · 2024-12-12T13:27:59Z

⚪ 2024-12-12 13:27:58 UTC Pre-commit check linux-x86_64-release-asan for a255b53 has started.
⚪ 2024-12-12 13:28:08 UTC Artifacts will be uploaded here
⚪ 2024-12-12 13:31:30 UTC ya make is running...
⚫ 2024-12-12 13:47:00 UTC Check cancelled

github-actions · 2024-12-12T13:28:00Z

⚪ 2024-12-12 13:28:00 UTC Pre-commit check linux-x86_64-relwithdebinfo for a255b53 has started.
⚪ 2024-12-12 13:28:11 UTC Artifacts will be uploaded here
⚪ 2024-12-12 13:31:34 UTC ya make is running...
⚫ 2024-12-12 13:47:00 UTC Check cancelled

github-actions · 2024-12-12T13:51:04Z

⚪ 2024-12-12 13:51:04 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7b0368c has started.
⚪ 2024-12-12 13:51:15 UTC Artifacts will be uploaded here
⚪ 2024-12-12 13:54:38 UTC ya make is running...
🟡 2024-12-12 15:38:33 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
32816	29890	0	4	2810	112

⚪ 2024-12-12 15:41:16 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-12-12 15:53:05 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
785 (only retried tests)	676	0	1	0	108

⚪ 2024-12-12 15:53:17 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-12-12 16:04:56 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
739 (only retried tests)	630	0	1	0	108

🟢 2024-12-12 16:05:05 UTC Build successful.
🟡 2024-12-12 16:05:26 UTC ydbd size 2.1 GiB changed* by +627.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `d78c86c`	merge: `7b0368c`	diff	diff %
ydbd size	2 256 263 536 Bytes	2 256 906 040 Bytes	+627.4 KiB	+0.028%
ydbd stripped size	483 943 952 Bytes	484 073 072 Bytes	+126.1 KiB	+0.027%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-12T13:51:12Z

⚪ 2024-12-12 13:51:12 UTC Pre-commit check linux-x86_64-release-asan for 7b0368c has started.
⚪ 2024-12-12 13:51:23 UTC Artifacts will be uploaded here
⚪ 2024-12-12 13:54:37 UTC ya make is running...
🟡 2024-12-12 15:26:30 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15317	15250	0	13	7	47

🟢 2024-12-12 15:27:44 UTC Build successful.
🟡 2024-12-12 15:28:08 UTC ydbd size 3.6 GiB changed* by +1012.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `d78c86c`	merge: `7b0368c`	diff	diff %
ydbd size	3 898 615 992 Bytes	3 899 652 800 Bytes	+1012.5 KiB	+0.027%
ydbd stripped size	1 366 738 000 Bytes	1 367 110 896 Bytes	+364.2 KiB	+0.027%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-13T07:19:58Z

⚪ 2024-12-13 07:19:57 UTC Pre-commit check linux-x86_64-relwithdebinfo for aa4d8c7 has started.
⚪ 2024-12-13 07:20:09 UTC Artifacts will be uploaded here
⚪ 2024-12-13 07:23:25 UTC ya make is running...
⚫ 2024-12-13 07:54:48 UTC Check cancelled

github-actions · 2024-12-13T07:20:15Z

⚪ 2024-12-13 07:20:14 UTC Pre-commit check linux-x86_64-release-asan for aa4d8c7 has started.
⚪ 2024-12-13 07:20:25 UTC Artifacts will be uploaded here
⚪ 2024-12-13 07:23:57 UTC ya make is running...
⚫ 2024-12-13 07:54:47 UTC Check cancelled

github-actions · 2024-12-13T07:56:16Z

⚪ 2024-12-13 07:56:16 UTC Pre-commit check linux-x86_64-release-asan for 655a86a has started.
⚪ 2024-12-13 07:56:34 UTC Artifacts will be uploaded here
⚪ 2024-12-13 08:00:02 UTC ya make is running...
⚫ 2024-12-13 08:30:01 UTC Check cancelled

github-actions · 2024-12-13T07:58:34Z

⚪ 2024-12-13 07:58:33 UTC Pre-commit check linux-x86_64-relwithdebinfo for 655a86a has started.
⚪ 2024-12-13 07:58:44 UTC Artifacts will be uploaded here
⚪ 2024-12-13 08:01:59 UTC ya make is running...
⚫ 2024-12-13 08:30:04 UTC Check cancelled

github-actions · 2024-12-13T08:34:09Z

⚪ 2024-12-13 08:34:08 UTC Pre-commit check linux-x86_64-relwithdebinfo for e869770 has started.
⚪ 2024-12-13 08:34:20 UTC Artifacts will be uploaded here
⚪ 2024-12-13 08:37:49 UTC ya make is running...
🟡 2024-12-13 10:16:17 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
32815	29875	0	2	2824	114

⚪ 2024-12-13 10:18:58 UTC ya make is running... (failed tests rerun, try 2)
🟡 2024-12-13 10:30:49 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
845 (only retried tests)	734	0	1	0	110

⚪ 2024-12-13 10:31:01 UTC ya make is running... (failed tests rerun, try 3)
🔴 2024-12-13 10:42:28 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
791 (only retried tests)	682	0	1	0	108

🟢 2024-12-13 10:42:38 UTC Build successful.
🟡 2024-12-13 10:42:59 UTC ydbd size 2.1 GiB changed* by +620.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `7782482`	merge: `e869770`	diff	diff %
ydbd size	2 256 566 000 Bytes	2 257 201 512 Bytes	+620.6 KiB	+0.028%
ydbd stripped size	484 005 744 Bytes	484 127 024 Bytes	+118.4 KiB	+0.025%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-13T08:34:20Z

⚪ 2024-12-13 08:34:19 UTC Pre-commit check linux-x86_64-release-asan for e869770 has started.
⚪ 2024-12-13 08:34:30 UTC Artifacts will be uploaded here
⚪ 2024-12-13 08:37:46 UTC ya make is running...
🟡 2024-12-13 10:08:21 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15318	15234	0	35	7	42

🟢 2024-12-13 10:09:36 UTC Build successful.
🟡 2024-12-13 10:10:03 UTC ydbd size 3.6 GiB changed* by +1007.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `7782482`	merge: `e869770`	diff	diff %
ydbd size	3 899 123 120 Bytes	3 900 154 256 Bytes	+1007.0 KiB	+0.026%
ydbd stripped size	1 366 840 144 Bytes	1 367 205 872 Bytes	+357.2 KiB	+0.027%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-13T15:10:40Z

⚪ 2024-12-13 15:10:39 UTC Pre-commit check linux-x86_64-relwithdebinfo for 79621c2 has started.
⚪ 2024-12-13 15:10:51 UTC Artifacts will be uploaded here
⚪ 2024-12-13 15:14:14 UTC ya make is running...
🟡 2024-12-13 16:43:50 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
32878	29932	0	11	2823	112

⚪ 2024-12-13 16:46:34 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-13 16:58:39 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
694 (only retried tests)	586	0	0	1	107

🟢 2024-12-13 16:58:49 UTC Build successful.
🟡 2024-12-13 16:59:06 UTC ydbd size 2.1 GiB changed* by +624.6 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `31afdb8`	merge: `79621c2`	diff	diff %
ydbd size	2 258 161 872 Bytes	2 258 801 456 Bytes	+624.6 KiB	+0.028%
ydbd stripped size	484 299 920 Bytes	484 425 264 Bytes	+122.4 KiB	+0.026%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-13T15:11:22Z

⚪ 2024-12-13 15:11:21 UTC Pre-commit check linux-x86_64-release-asan for 79621c2 has started.
⚪ 2024-12-13 15:11:55 UTC Artifacts will be uploaded here
⚪ 2024-12-13 15:15:47 UTC ya make is running...
🟡 2024-12-13 16:48:56 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
15326	15247	0	18	16	45

🟢 2024-12-13 16:50:14 UTC Build successful.
🟡 2024-12-13 16:50:39 UTC ydbd size 3.6 GiB changed* by +1007.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `31afdb8`	merge: `79621c2`	diff	diff %
ydbd size	3 902 464 424 Bytes	3 903 495 568 Bytes	+1007.0 KiB	+0.026%
ydbd stripped size	1 367 972 592 Bytes	1 368 338 320 Bytes	+357.2 KiB	+0.027%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-16T08:59:18Z

⚪ 2024-12-16 08:59:18 UTC Pre-commit check linux-x86_64-release-asan for 4e9af25 has started.
⚪ 2024-12-16 08:59:31 UTC Artifacts will be uploaded here
⚪ 2024-12-16 09:02:59 UTC ya make is running...
🟡 2024-12-16 10:44:32 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
13976	13877	0	33	22	44

🟢 2024-12-16 10:45:43 UTC Build successful.
🟡 2024-12-16 10:46:09 UTC ydbd size 3.6 GiB changed* by +1017.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `b0b1457`	merge: `4e9af25`	diff	diff %
ydbd size	3 890 739 688 Bytes	3 891 781 632 Bytes	+1017.5 KiB	+0.027%
ydbd stripped size	1 362 162 448 Bytes	1 362 534 832 Bytes	+363.7 KiB	+0.027%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-16T08:59:22Z

⚪ 2024-12-16 08:59:21 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4e9af25 has started.
⚪ 2024-12-16 08:59:33 UTC Artifacts will be uploaded here
⚪ 2024-12-16 09:02:47 UTC ya make is running...
🟡 2024-12-16 10:32:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
30961	28023	0	3	2824	111

⚪ 2024-12-16 10:35:15 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-16 10:46:45 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
159 (only retried tests)	51	0	0	0	108

🟢 2024-12-16 10:46:55 UTC Build successful.
🟡 2024-12-16 10:47:13 UTC ydbd size 2.1 GiB changed* by +627.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `b0b1457`	merge: `4e9af25`	diff	diff %
ydbd size	2 251 866 360 Bytes	2 252 508 736 Bytes	+627.3 KiB	+0.029%
ydbd stripped size	482 574 224 Bytes	482 700 144 Bytes	+123.0 KiB	+0.026%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-17T10:15:12Z

⚪ 2024-12-17 10:15:11 UTC Pre-commit check linux-x86_64-release-asan for 73f3608 has started.
⚪ 2024-12-17 10:15:23 UTC Artifacts will be uploaded here
⚪ 2024-12-17 10:18:37 UTC ya make is running...
🟡 2024-12-17 11:55:41 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14020	13947	0	19	6	48

🟢 2024-12-17 11:56:52 UTC Build successful.
🟡 2024-12-17 11:57:16 UTC ydbd size 3.6 GiB changed* by +1013.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `99fdf00`	merge: `73f3608`	diff	diff %
ydbd size	3 895 651 712 Bytes	3 896 689 352 Bytes	+1013.3 KiB	+0.027%
ydbd stripped size	1 363 547 536 Bytes	1 363 916 048 Bytes	+359.9 KiB	+0.027%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-12-17T10:15:40Z

⚪ 2024-12-17 10:15:39 UTC Pre-commit check linux-x86_64-relwithdebinfo for 73f3608 has started.
⚪ 2024-12-17 10:15:51 UTC Artifacts will be uploaded here
⚪ 2024-12-17 10:19:04 UTC ya make is running...
🟡 2024-12-17 11:50:26 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
31019	28058	0	11	2832	118

⚪ 2024-12-17 11:53:30 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-12-17 12:05:40 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
182 (only retried tests)	70	0	0	1	111

🟢 2024-12-17 12:05:47 UTC Build successful.
🟡 2024-12-17 12:06:09 UTC ydbd size 2.1 GiB changed* by +623.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `99fdf00`	merge: `73f3608`	diff	diff %
ydbd size	2 254 662 928 Bytes	2 255 301 208 Bytes	+623.3 KiB	+0.028%
ydbd stripped size	483 010 352 Bytes	483 132 144 Bytes	+118.9 KiB	+0.025%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions bot added the improvement label Dec 12, 2024

snaury force-pushed the feature-11561-ds-state-migration branch from 8ba8d3d to a600110 Compare December 12, 2024 13:46

snaury force-pushed the feature-11561-ds-state-migration branch from a600110 to 9929942 Compare December 13, 2024 07:16

snaury force-pushed the feature-11561-ds-state-migration branch from 9929942 to c986549 Compare December 13, 2024 07:54

snaury marked this pull request as ready for review December 13, 2024 08:25

snaury requested a review from azevaykin December 13, 2024 08:25

snaury force-pushed the feature-11561-ds-state-migration branch from c986549 to 53d346a Compare December 13, 2024 08:29

snaury self-assigned this Dec 13, 2024

snaury force-pushed the feature-11561-ds-state-migration branch from 53d346a to 233d6aa Compare December 13, 2024 15:07

snaury force-pushed the feature-11561-ds-state-migration branch from 233d6aa to 6b4ccf9 Compare December 16, 2024 08:54

azevaykin previously approved these changes Dec 16, 2024

View reviewed changes

Try to preserve in-memory state (locks in particular) across restarts

a99b0f8

snaury dismissed azevaykin’s stale review via a99b0f8 December 17, 2024 10:11

snaury force-pushed the feature-11561-ds-state-migration branch from 6b4ccf9 to a99b0f8 Compare December 17, 2024 10:11

snaury requested a review from azevaykin December 17, 2024 10:12

snaury enabled auto-merge (squash) December 17, 2024 10:12

azevaykin approved these changes Dec 17, 2024

View reviewed changes

snaury merged commit 02fdbe3 into ydb-platform:main Dec 17, 2024
10 checks passed

snaury deleted the feature-11561-ds-state-migration branch December 17, 2024 12:44

This was referenced Dec 18, 2024

YQ-3975 RD fixed fault for parsing errors without filter #12707

Merged

YQ-3970 RD fixed memory leak in filters #12739

Merged

niksaveliev mentioned this pull request Dec 25, 2024

Kafka connect distributed #12960

Merged

GrigoriyPA mentioned this pull request Dec 25, 2024

Fixed empty error from CMS create tenant request #12977

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Try to preserve in-memory state (locks in particular) across restarts #12567

Try to preserve in-memory state (locks in particular) across restarts #12567

Uh oh!

snaury commented Dec 12, 2024

github-actions bot commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 16, 2024 •

edited

Loading

github-actions bot commented Dec 16, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

Uh oh!

Labels

2 participants

Try to preserve in-memory state (locks in particular) across restarts #12567

Try to preserve in-memory state (locks in particular) across restarts #12567

Uh oh!

Conversation

snaury commented Dec 12, 2024

Changelog entry

Changelog category

Additional information

github-actions bot commented Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 13, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 16, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

github-actions bot commented Dec 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Labels

2 participants

github-actions bot commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 12, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 13, 2024 •

edited

Loading

github-actions bot commented Dec 16, 2024 •

edited

Loading

github-actions bot commented Dec 16, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 17, 2024 •

edited

Loading