Skip to content

Avoid persistent and in-memory tx status getting out-of-sync #14704

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 18, 2025
Merged

Avoid persistent and in-memory tx status getting out-of-sync #14704

merged 1 commit into from
Feb 18, 2025

Conversation

@snaury
Copy link
Member

@snaury snaury commented Feb 17, 2025

Changelog entry

...

Changelog category

  • Not for changelog (changelog entry is not required)

Description for reviewers

It was discovered that in certain situation tx status parts and in-memory state may diverge, allowing transaction ids to be reused, which would unexpectedly become committed or removed after a restart. Make sure in-memory state doesn't diverge from the state that could be restored from disk on restart.

Fixes KIKIMR-22538.
@snaury snaury self-assigned this Feb 17, 2025
@github-actions
Copy link

github-actions bot commented Feb 17, 2025

🟢 2025-02-17 15:51:23 UTC The validation of the Pull Request description is successful.
@github-actions
Copy link

github-actions bot commented Feb 17, 2025 •
edited
Loading

⚪ 2025-02-17 15:52:00 UTC Pre-commit check linux-x86_64-relwithdebinfo for d1998f2 has started.
⚪ 2025-02-17 15:52:13 UTC Artifacts will be uploaded here
⚪ 2025-02-17 15:55:13 UTC ya make is running...
🟡 2025-02-17 17:27:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27729 25126 0 3 2466 134

⚪ 2025-02-17 17:30:00 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-02-17 17:43:24 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
224 (only retried tests) 98 0 0 0 126

🟢 2025-02-17 17:43:31 UTC Build successful.
🟢 2025-02-17 17:43:51 UTC ydbd size 2.1 GiB changed* by +85.5 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: ebacf45 merge: d1998f2 diff diff %
ydbd size 2 236 295 200 Bytes 2 236 382 800 Bytes +85.5 KiB +0.004%
ydbd stripped size 473 853 496 Bytes 473 877 688 Bytes +23.6 KiB +0.005%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
@github-actions
Copy link

github-actions bot commented Feb 17, 2025 •
edited
Loading

⚪ 2025-02-17 15:53:53 UTC Pre-commit check linux-x86_64-release-asan for d1998f2 has started.
⚪ 2025-02-17 15:54:20 UTC Artifacts will be uploaded here
⚪ 2025-02-17 15:58:01 UTC ya make is running...
🟡 2025-02-17 17:26:03 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13550 13461 0 34 19 36

⚪ 2025-02-17 17:27:30 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-02-17 17:39:24 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
133 (only retried tests) 95 0 0 5 33

🟢 2025-02-17 17:39:36 UTC Build successful.
🟡 2025-02-17 17:40:04 UTC ydbd size 3.6 GiB changed* by +109.9 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: ebacf45 merge: d1998f2 diff diff %
ydbd size 3 890 108 904 Bytes 3 890 221 416 Bytes +109.9 KiB +0.003%
ydbd stripped size 1 361 740 672 Bytes 1 361 773 536 Bytes +32.1 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
@snaury snaury force-pushed the bugfix-KIKIMR-22538-txstatus-compaction branch from dc176f7 to b7d11cd Compare February 18, 2025 07:44
@github-actions
Copy link

github-actions bot commented Feb 18, 2025 •
edited
Loading

⚪ 2025-02-18 07:46:56 UTC Pre-commit check linux-x86_64-relwithdebinfo for 06bace0 has started.
⚪ 2025-02-18 08:01:22 UTC Artifacts will be uploaded here
⚪ 2025-02-18 08:04:37 UTC ya make is running...
🟡 2025-02-18 09:24:49 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27748 25138 0 7 2469 134

⚪ 2025-02-18 09:27:07 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-02-18 09:54:38 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
232 (only retried tests) 101 0 1 2 128

⚪ 2025-02-18 09:54:47 UTC ya make is running... (failed tests rerun, try 3)
🔴 2025-02-18 10:11:12 UTC Some tests failed, follow the links below.

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
193 (only retried tests) 63 0 1 0 129

🟢 2025-02-18 10:11:22 UTC Build successful.
🟢 2025-02-18 10:11:47 UTC ydbd size 2.1 GiB changed* by +88.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: 22bee29 merge: 06bace0 diff diff %
ydbd size 2 236 147 032 Bytes 2 236 237 968 Bytes +88.8 KiB +0.004%
ydbd stripped size 473 902 008 Bytes 473 928 120 Bytes +25.5 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
@github-actions
Copy link

github-actions bot commented Feb 18, 2025 •
edited
Loading

⚪ 2025-02-18 07:47:54 UTC Pre-commit check linux-x86_64-release-asan for 06bace0 has started.
⚪ 2025-02-18 07:48:07 UTC Artifacts will be uploaded here
⚪ 2025-02-18 07:51:08 UTC ya make is running...
🟡 2025-02-18 09:23:18 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13560 13465 0 31 27 37

⚪ 2025-02-18 09:24:28 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-02-18 09:36:37 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
147 (only retried tests) 103 0 3 8 33

⚪ 2025-02-18 09:36:45 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-02-18 09:55:07 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
74 (only retried tests) 33 0 3 7 31

🟢 2025-02-18 09:55:16 UTC Build successful.
🟡 2025-02-18 09:55:50 UTC ydbd size 3.6 GiB changed* by +108.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 22bee29 merge: 06bace0 diff diff %
ydbd size 3 890 200 496 Bytes 3 890 311 152 Bytes +108.1 KiB +0.003%
ydbd stripped size 1 361 992 704 Bytes 1 362 028 064 Bytes +34.5 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
@naspirato naspirato added the rebase-and-check Rebase PR with the current base branch and check label Feb 18, 2025
@github-actions github-actions bot removed the rebase-and-check Rebase PR with the current base branch and check label Feb 18, 2025
@github-actions
Copy link

github-actions bot commented Feb 18, 2025

⚪ 2025-02-18 10:55:10 UTC Pre-commit check linux-x86_64-release-asan for eea9171 has started.
@github-actions
Copy link

github-actions bot commented Feb 18, 2025 •
edited
Loading

⚪ 2025-02-18 10:56:50 UTC Pre-commit check linux-x86_64-relwithdebinfo for eea9171 has started.
⚪ 2025-02-18 10:57:02 UTC Artifacts will be uploaded here
⚪ 2025-02-18 11:00:13 UTC ya make is running...
🟡 2025-02-18 12:32:10 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27748 25139 0 1 2471 137

⚪ 2025-02-18 12:34:30 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-02-18 12:47:50 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
221 (only retried tests) 89 0 0 3 129

🟢 2025-02-18 12:47:58 UTC Build successful.
🟡 2025-02-18 12:48:22 UTC ydbd size 2.1 GiB changed* by +331.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: 0a1a524 merge: eea9171 diff diff %
ydbd size 2 236 147 032 Bytes 2 236 486 528 Bytes +331.5 KiB +0.015%
ydbd stripped size 473 902 008 Bytes 473 959 352 Bytes +56.0 KiB +0.012%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
@snaury snaury marked this pull request as ready for review February 18, 2025 12:42
@snaury snaury requested a review from azevaykin February 18, 2025 12:43
@snaury snaury force-pushed the bugfix-KIKIMR-22538-txstatus-compaction branch from b7d11cd to de6aa06 Compare February 18, 2025 13:04
@github-actions
Copy link

github-actions bot commented Feb 18, 2025 •
edited
Loading

⚪ 2025-02-18 13:10:28 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5bba660 has started.
⚪ 2025-02-18 13:10:41 UTC Artifacts will be uploaded here
⚪ 2025-02-18 13:13:42 UTC ya make is running...
⚫ 2025-02-18 13:40:42 UTC Check cancelled
@github-actions
Copy link

github-actions bot commented Feb 18, 2025 •
edited
Loading

⚪ 2025-02-18 13:14:20 UTC Pre-commit check linux-x86_64-release-asan for 5bba660 has started.
⚪ 2025-02-18 13:14:33 UTC Artifacts will be uploaded here
⚪ 2025-02-18 13:17:45 UTC ya make is running...
⚫ 2025-02-18 13:40:38 UTC Check cancelled
azevaykin
azevaykin reviewed Feb 18, 2025
View reviewed changes
@snaury snaury force-pushed the bugfix-KIKIMR-22538-txstatus-compaction branch from de6aa06 to 7f63742 Compare February 18, 2025 13:40
@snaury snaury requested a review from azevaykin February 18, 2025 13:40
@github-actions
Copy link

github-actions bot commented Feb 18, 2025 •
edited
Loading

⚪ 2025-02-18 13:42:11 UTC Pre-commit check linux-x86_64-release-asan for 5a9e6e0 has started.
⚪ 2025-02-18 13:42:43 UTC Artifacts will be uploaded here
⚪ 2025-02-18 13:46:25 UTC ya make is running...
🟡 2025-02-18 15:10:48 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
13560 13498 0 16 11 35

⚪ 2025-02-18 15:12:13 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-02-18 15:27:18 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
97 (only retried tests) 59 0 0 6 32

🟢 2025-02-18 15:27:27 UTC Build successful.
🟡 2025-02-18 15:27:55 UTC ydbd size 3.6 GiB changed* by +104.1 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash main: d1f5b91 merge: 5a9e6e0 diff diff %
ydbd size 3 891 711 472 Bytes 3 891 818 032 Bytes +104.1 KiB +0.003%
ydbd stripped size 1 362 428 448 Bytes 1 362 459 712 Bytes +30.5 KiB +0.002%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
@github-actions
Copy link

github-actions bot commented Feb 18, 2025 •
edited
Loading

⚪ 2025-02-18 13:44:32 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5a9e6e0 has started.
⚪ 2025-02-18 13:44:45 UTC Artifacts will be uploaded here
⚪ 2025-02-18 13:47:45 UTC ya make is running...
🟡 2025-02-18 15:18:11 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
27752 25144 0 1 2470 137

⚪ 2025-02-18 15:21:18 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-02-18 15:42:37 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
218 (only retried tests) 87 0 0 0 131

🟢 2025-02-18 15:42:44 UTC Build successful.
🟢 2025-02-18 15:43:09 UTC ydbd size 2.1 GiB changed* by +88.8 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash main: d1f5b91 merge: 5a9e6e0 diff diff %
ydbd size 2 236 933 728 Bytes 2 237 024 664 Bytes +88.8 KiB +0.004%
ydbd stripped size 474 020 440 Bytes 474 046 552 Bytes +25.5 KiB +0.006%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation
@snaury snaury merged commit eb1eed1 into ydb-platform:main Feb 18, 2025
12 checks passed
@snaury snaury deleted the bugfix-KIKIMR-22538-txstatus-compaction branch February 18, 2025 17:26
@snaury snaury restored the bugfix-KIKIMR-22538-txstatus-compaction branch February 19, 2025 08:40
snaury added a commit that referenced this pull request Feb 19, 2025
@snaury
Revert "Avoid persistent and in-memory tx status getting out-of-sync (#…
862db56 
…14704)" This reverts commit eb1eed1.
@snaury snaury mentioned this pull request Feb 19, 2025
Merged
@snaury snaury deleted the bugfix-KIKIMR-22538-txstatus-compaction branch February 19, 2025 12:40
This was referenced Feb 19, 2025
Merged
Closed
Merged
@serbel324 serbel324 mentioned this pull request Feb 25, 2025
Merged
lberserq pushed a commit to lberserq/ydb that referenced this pull request Mar 3, 2025
@snaury
Avoid persistent and in-memory tx status getting out-of-sync (ydb-pla…
41b97f6 
…tform#14704)
@GrigoriyPA GrigoriyPA mentioned this pull request Mar 4, 2025
Closed
This was referenced Mar 10, 2025
Merged
Merged
blinkov pushed a commit that referenced this pull request Mar 21, 2025
@snaury @blinkov
Avoid persistent and in-memory tx status getting out-of-sync (#14704)
f6139d5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

not-for-changelog

3 participants

@snaury @azevaykin @naspirato