Skip to content

Conversation

@ijon
Copy link
Collaborator

@ijon ijon commented Aug 6, 2025

Merge from main:

Currently, drop-extsubdomain operations may not completely remove dedicated databases from the system:

  • system tablets of the tenant may remain in the root hive
  • traces of database paths and subdomain objects may persist in the root schemeshard

Root cause: The cleanup procedure depends on the tenant hive's existence - delete-tablet requests are sent to the tenant hive for redirection to the root hive. This works for most databases but fails with dedicated databases, where the tenant hive cannot outlive drop-extsubdomain operations, and there's no guarantee that tenant nodes and tablets will be available during deletion.

Changes:

  • Implement quick fix for subdomain cleanup by directly sending system tablets delete-tablet requests to the root hive (only for dedicated databases)
  • Add unit and reboot tests to verify proper extsubdomain cleanup
  • Fix aggressive reconnect loop in coordinator's mediator_queue
  • Fix handling of TEvSchemeShard::TEvSyncTenantSchemeShard in schemeshard

Closes #19842.

Changelog entry

Fix issue where dedicated database deletion may leave database system tablets improperly cleaned.

Changelog category

  • Bugfix
@github-actions
Copy link

github-actions bot commented Aug 6, 2025

2025-08-06 15:18:42 UTC Pre-commit check linux-x86_64-release-asan for 3ca24e1 has started.
2025-08-06 15:19:43 UTC Check cancelled

@github-actions
Copy link

github-actions bot commented Aug 6, 2025

🟢 2025-08-06 15:19:11 UTC The validation of the Pull Request description is successful.

@ijon ijon force-pushed the merge/25-1-3/quickfix-extsubdomain-drop-missing-system-tablets branch 2 times, most recently from 95ef961 to dee0cd9 Compare August 6, 2025 16:49
@github-actions

This comment was marked as outdated.

@github-actions

This comment was marked as outdated.

@ydb-platform ydb-platform deleted a comment from github-actions bot Aug 6, 2025
@ydb-platform ydb-platform deleted a comment from github-actions bot Aug 6, 2025
@ijon ijon linked an issue Aug 6, 2025 that may be closed by this pull request
ijon added 2 commits August 7, 2025 14:07
…hive (ydb-platform#21377) - Implement quick fix for subdomain cleanup by directly sending system tablets delete-tablet requests to the root hive (only for dedicated databases) - Add unit and reboot tests to verify proper extsubdomain cleanup - Fix aggressive reconnect loop in coordinator's mediator_queue - Fix handling of TEvSchemeShard::TEvSyncTenantSchemeShard in schemeshard
@ijon ijon force-pushed the merge/25-1-3/quickfix-extsubdomain-drop-missing-system-tablets branch from dee0cd9 to 11640a8 Compare August 7, 2025 11:07
@github-actions
Copy link

github-actions bot commented Aug 7, 2025

2025-08-07 11:13:07 UTC Pre-commit check linux-x86_64-release-asan for dc311dc has started.
2025-08-07 11:13:33 UTC Artifacts will be uploaded here
2025-08-07 11:17:12 UTC ya make is running...
🟡 2025-08-07 12:27:52 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
12683 12486 0 53 110 34

2025-08-07 12:29:04 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-08-07 12:55:58 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
254 (only retried tests) 197 0 6 16 35

2025-08-07 12:56:11 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-08-07 13:18:01 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
86 (only retried tests) 41 0 4 7 34

🟢 2025-08-07 13:18:10 UTC Build successful.
🟡 2025-08-07 13:18:42 UTC ydbd size 3.7 GiB changed* by +119.6 KiB, which is >= 100.0 KiB vs stable-25-1-3: Warning

ydbd size dash stable-25-1-3: 212c3eb merge: dc311dc diff diff %
ydbd size 4 021 544 832 Bytes 4 021 667 312 Bytes +119.6 KiB +0.003%
ydbd stripped size 1 402 629 432 Bytes 1 402 672 728 Bytes +42.3 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions
Copy link

github-actions bot commented Aug 7, 2025

2025-08-07 11:13:15 UTC Pre-commit check linux-x86_64-relwithdebinfo for dc311dc has started.
2025-08-07 11:13:29 UTC Artifacts will be uploaded here
2025-08-07 11:16:27 UTC ya make is running...
🟡 2025-08-07 12:23:01 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
19826 18534 0 3 1243 46

2025-08-07 12:55:04 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-08-07 13:16:59 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
104 (only retried tests) 60 0 0 4 40

🟢 2025-08-07 13:17:07 UTC Build successful.
🟢 2025-08-07 13:17:29 UTC ydbd size 2.2 GiB changed* by +55.8 KiB, which is < 100.0 KiB vs stable-25-1-3: OK

ydbd size dash stable-25-1-3: 212c3eb merge: dc311dc diff diff %
ydbd size 2 310 337 024 Bytes 2 310 394 152 Bytes +55.8 KiB +0.002%
ydbd stripped size 485 378 656 Bytes 485 392 672 Bytes +13.7 KiB +0.003%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@ijon ijon marked this pull request as ready for review August 7, 2025 13:22
@ijon ijon requested a review from a team as a code owner August 7, 2025 13:22
@ijon ijon merged commit 460774a into ydb-platform:stable-25-1-3 Aug 7, 2025
12 checks passed
@ijon ijon deleted the merge/25-1-3/quickfix-extsubdomain-drop-missing-system-tablets branch August 7, 2025 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

3 participants