Bug#119037 Fix OOM during MySQL upgrade with large number of tables

[akshatnehra]

Description

A unbounded memory usage issue has been identified in the check_table_funs function, introduced in commit 7c0dda9. This function is responsible for validating SQL functions within table definitions during a database upgrade. It occurs because tables that are opened for validation are not subsequently closed. For databases with a very large number of tables, this leads to uncontrolled memory growth, eventually causing the MySQL server to be terminated by the operating system's OOM (Out of Memory) killer.

The issue stems from the fact that check_table_funs iterates through all tables and calls open_table for each one, but fails to call a corresponding function to close the tables and release the associated memory. This behavior was not present in previous, and as a result, upgrades on databases with a high table count were significantly faster and consumed a stable amount of memory.

How to recreate the issue

The bug can be reproduced by performing an upgrade from MySQL 8.0.40 to 8.0.42 on a server with a schema containing a large number of tables (e.g., ~1 million).

1. Set up a MySQL 8.0.40 server.
2. Create a schema and populate it with approximately 1 million tables.
3. Attempt to upgrade the server to MySQL 8.0.42.

During the upgrade process, memory consumption will steadily increase until the server is terminated by the OOM killer. As a point of comparison, upgrading the same database from 8.0.40 to 8.0.41 completes successfully in a much shorter time and without excessive memory usage.

Fix

The provided patch resolves the unbounded memory usage issue by ensuring that each table is closed immediately after its validation is complete. This is achieved by adding a call to close_thread_table within the loop in the check_table_funs function, after the open_table call. This change correctly implements the original feature's intent, which was to perform a temporary "try to open" check without lasting side effects. The original code was missing this cleanup step on its success path, leading to uncontrolled memory growth.

This fix is safe. The entire upgrade check runs in an isolated bootstrap thread that starts with no open tables. Furthermore, the operation is precise: open_table adds a handle to the head of a list, and close_thread_table removes that exact same handle. Since open_table does not recursively open handles for dependencies like triggers or foreign keys, this single close should be sufficient to release all resources acquired in the loop, preventing the OOM error and allowing the upgrade to complete successfully.

Testing

The fix has been verified by running the main.dd_upgrade_36890891 MTR test. Furthermore, the scenario described in the bug report was recreated. Without the patch, the upgrade failed due to an OOM error. With the patch applied, the upgrade to 8.0.42 completed successfully without consuming an excessive amount of memory, confirming the effectiveness of the fix.

---

This contribution is under the OCA signed by Amazon and covering submissions to the MySQL project.

[mysql-oca-bot]

Hi, thank you for your contribution. Please confirm this code is submitted under the terms of the OCA (Oracle's Contribution Agreement) you have previously signed by cutting and pasting the following text as a comment:
"I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it."
Thanks

[akshatnehra]

I confirm the code being submitted is offered under the terms of the OCA, and that I am authorized to contribute it.

[mysql-oca-bot]

Hi, thank you for your contribution. Your code has been assigned to an internal queue. Please follow
bug http://bugs.mysql.com/bug.php?id=119037 for updates.
Thanks