Replaced `CREATE TEMPORARY TABLE ... LIKE`with different approach #36022

michalbiarda · 2022-08-25T10:39:40Z

Description (*)

Magento's MySQL adapter uses CREATE TEMPORARY TABLE ... LIKE command in \Magento\Framework\DB\Adapter\Pdo\Mysql::createTemporaryTableLike.

Newer versions of MySQL (8+) have problems with CREATE TEMPORARY TABLE ... LIKE, as you can read in the official documentation (https://dev.mysql.com/doc/refman/8.0/en/create-temporary-table.html):

You cannot use CREATE TEMPORARY TABLE ... LIKE to create an empty table based on the definition of a table that resides in the mysql tablespace, InnoDB system tablespace (innodb_system), or a general tablespace.

The documentation suggests to use CREATE TEMPORARY TABLE ... SELECT * FROM ... LIMIT 0 instead, but such command IS NOT an exact equivalent. It will create temporary table with the same fields, but will not create indexes. Without indexes performance of reindexing big amount of data is absurdly slow.

I suggest to build CREATE TEMPORARY TABLE without LIKE and without SELECT * FROM. Here's my idea:

Firstly use SHOW CREATE TABLE to get query for creation of source table (including indexes creation), and then modify it with PHP: change CREATE TABLE to CREATE TEMPORARY TABLE, add IF NOT EXISTS if necessary, remove TABLESPACE definition if it exists, and replace source table name with temporary table name.

Manual testing scenarios (*)

Install Magento 2.4 on a server with MySQL 8.
Apply sample data.
Run bin/magento index:reindex.
Check data on frontend.

Contribution checklist (*)

Pull request has a meaningful description of its purpose
All commits are accompanied by meaningful commit messages
All new or changed code is covered with unit/integration tests (if applicable)
README.md files for modified modules are updated and included in the pull request if any README.md predefined sections require an update
All automated tests passed successfully (all builds are green)

Resolved issues:

resolves [Issue] Replaced CREATE TEMPORARY TABLE ... LIKEwith different approach #37926: Replaced CREATE TEMPORARY TABLE ... LIKEwith different approach

…at doesn't conflict with MySQL 8

m2-assistant · 2022-08-25T10:39:42Z

Hi @michalbiarda. Thank you for your contribution
Here are some useful tips how you can test your changes using Magento test environment.
Add the comment under your pull request to deploy test or vanilla Magento instance:

@magento give me test instance - deploy test instance based on PR changes
@magento give me 2.4-develop instance - deploy vanilla Magento instance

❗ Automated tests can be triggered manually with an appropriate comment:

@magento run all tests - run or re-run all required tests against the PR changes
@magento run <test-build(s)> - run or re-run specific test build(s)
For example: @magento run Unit Tests

<test-build(s)> is a comma-separated list of build names. Allowed build names are:

Database Compare
Functional Tests CE
Functional Tests EE,
Functional Tests B2B
Integration Tests
Magento Health Index
Sample Data Tests CE
Sample Data Tests EE
Sample Data Tests B2B
Static Tests
Unit Tests
WebAPI Tests
Semantic Version Checker

You can find more information about the builds here

ℹ️ Run only required test builds during development. Run all test builds before sending your pull request for review.

For more details, review the Magento Contributor Guide documentation.

⚠️ According to the Magento Contribution requirements, all Pull Requests must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.

🕙 You can find the schedule on the Magento Community Calendar page.

📞 The triage of Pull Requests happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

✏️ Feel free to post questions/proposals/feedback related to the Community Contributions Triage process to the corresponding Slack Channel

michalbiarda · 2022-08-25T10:44:27Z

@magento give me test instance

magento-deployment-service · 2022-08-25T10:44:31Z

Hi @michalbiarda. Thank you for your request. I'm working on Magento instance for you.

magento-deployment-service · 2022-08-25T10:48:29Z

Hi @michalbiarda, here is your Magento Instance: https://eb2c38a2f68947025bfccecf2d856c46.instances.magento-community.engineering
Admin access: https://eb2c38a2f68947025bfccecf2d856c46.instances.magento-community.engineering/admin_7f99
Login: 8e47b66d
Password: fd846693815e

michalbiarda · 2022-08-25T10:51:03Z

@magento run all tests

magento-automated-testing · 2022-08-25T10:51:09Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

kandy · 2022-08-25T14:25:53Z

lib/internal/Magento/Framework/DB/Adapter/Pdo/Mysql.php

 $temporaryTable = $this->quoteIdentifier($this->_getTableName($temporaryTableName));
 $originTable = $this->quoteIdentifier($this->_getTableName($originTableName));
- $sql = sprintf('CREATE TEMPORARY TABLE %s %s LIKE %s', $ifNotExistsSql, $temporaryTable, $originTable);
+ $originCreate = $this->fetchPairs(sprintf('SHOW CREATE TABLE %s', $originTable));


Suggested change

$originCreate = $this->fetchPairs(sprintf('SHOW CREATE TABLE %s', $originTable));

$originCreate = $this->fetchOne("SHOW CREATE TABLE {$originTable}");

You can use fetchOne to get result
Also sprintf is unnecessary

@kandy Unfortunately I can't use fetchOne because SHOW CREATE TABLE returns two fields. First is table name, the second is SQL query.

I like you suggestion about sprintf, though 👍.

michalbiarda · 2022-08-25T14:38:36Z

@kandy I'm converting this PR to draft. I'll apply your suggestion about sprintf, but I will also fix one more thing reported by someone else in the meantime.

…nflict with MySQL 8

michalbiarda · 2022-08-26T13:58:46Z

@magento run all tests

magento-automated-testing · 2022-08-26T13:58:51Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please re-request them if they don't show in a reasonable amount of time.

engcom-Hotel · 2023-06-08T12:50:31Z

@magento run Static Tests, Functional Tests EE

magento-automated-testing · 2023-06-08T12:50:39Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

engcom-Hotel · 2023-06-09T09:29:00Z

Hello @michalbiarda,

Thanks for the contribution!

According to the below comment here, it seems that you are still working on it.

@kandy I'm converting this PR to draft. I'll apply your suggestion about sprintf, but I will also fix one more thing reported by someone else in the meantime.

Can you please confirm, if the PR is ready for review?

Thanks

michalbiarda · 2023-06-09T09:54:56Z

Hello @engcom-Hotel,

After that comment I added something to my change, and marked this PR as ready for review 10 months ago 😅. I confirm it's still ready for review 😄.

engcom-Lima · 2023-06-09T11:08:45Z

@magento run Functional Tests EE

magento-automated-testing · 2023-06-09T11:08:54Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

engcom-Hotel · 2023-07-13T10:00:14Z

Hello @michalbiarda,

Thanks for the reply!

Hope you had a great vacation :)

After changing the innodb_file_per_table to OFF, the issue has been reproducible:

Hence we are proceeding further with this PR.

Thanks

engcom-Hotel · 2023-07-14T10:02:34Z

✔️ QA Passed

Preconditions:

SET innodb_file_per_table to OFF.

Manual testing scenario:

Install Magento 2.4 on a server with MySQL 8.
Generate data using the performance toolkit provided by Magento. Please refer to this link for this same.
Run bin/magento index:reindex.
Check data on front end.

Actual Result: ✔️

Reindex command should run properly without any errors and frontend is accessible with categories and products.

After: ✔️

Before: ✖️

Some functional tests are still failing hence moving this PR into extended testing.

Thanks

engcom-Echo · 2023-07-17T04:34:22Z

@magento run Functional Tests EE

magento-automated-testing · 2023-07-17T04:34:31Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

engcom-Echo · 2023-07-17T06:34:39Z

Functional Tests EE failures are different on last two run on same commit.
Run1

Run2

Hence moving to Merge In progress

engcom-Charlie · 2023-08-25T09:51:21Z

@magento run Functional Tests EE

magento-automated-testing · 2023-08-25T09:51:26Z

The requested builds are added to the queue. You should be able to see them here within a few minutes. Please message the #magento-devops slack channel if they don't show in a reasonable amount of time and a representative will look into any issues.

engcom-Hotel · 2023-08-28T10:26:04Z

@magento create issue

engcom-Hotel · 2023-08-28T11:31:56Z

Moving back this PR to Merge in progress

kennylawrence · 2024-11-04T00:29:51Z

...sooooo..... what exactly is the solution? I currently have to remember to modify the mysql pdo adapter file (vendor/magento/framework/DB/Adapter/Pdo/Mysql.php) every time I update with any new Magento releases. This has been going on for at least four years, ever since MySQL v8. I host at MageMojo/Webscale, and at the moment the following links to their "official" fix for this at the following page whenever I forget and end up creating a new ticket because indexing fails: https://github.com/magemojo/m2-patches/wiki

Is there going to be a more permanent solution? Or is something wrong with the configuration at one of the largest Magento hosting environments (or possibly just my old config is out of date) ? From this issue thread, I wasn't able to grasp what exactly is the current status of the issue (wrong config? code?). Please can we have the final word on this?

kennylawrence · 2024-11-04T01:02:01Z

OK - I just realized that these kinds of fixes aren't added to security releases. I see in 2.4.7 the file has been updated according to the different approach mentioned above, while it does not exist in 2.4.6-p8. I will copy the relevant function over to my file. I am happy this is finally being put to rest!

Replaced CREATE TEMPORARY TABLE ... LIKE with different approach th…

c47f5c5

…at doesn't conflict with MySQL 8

kandy reviewed Aug 25, 2022

View reviewed changes

m2-community-project bot assigned kandy Aug 25, 2022

m2-community-project bot added the Progress: review label Aug 25, 2022

michalbiarda marked this pull request as draft August 25, 2022 14:37

Removed TABLESPACE definition from CREATE TEMPORARY TABLE to not co…

a1f1ee3

…nflict with MySQL 8

michalbiarda marked this pull request as ready for review August 26, 2022 13:55

engcom-Hotel added the Priority: P1 Once P0 defects have been fixed, a defect having this priority is the next candidate for fixing. label Aug 26, 2022

m2-community-project bot added Progress: pending review and removed Progress: review labels Dec 8, 2022

nrkapoor assigned chernenm May 28, 2023

m2-community-project bot added Progress: review and removed Progress: pending review labels May 28, 2023

engcom-Lima added the Triage: Dev.Experience Issue related to Developer Experience and needs help with Triage to Confirm or Reject it label Jun 9, 2023

engcom-Hotel self-requested a review June 12, 2023 10:37

m2-community-project bot added Progress: review and removed Progress: on hold labels Jul 13, 2023

engcom-Hotel approved these changes Jul 13, 2023

View reviewed changes

m2-community-project bot assigned engcom-Hotel Jul 13, 2023

m2-community-project bot added Progress: ready for testing Progress: testing in progress and removed Progress: review Progress: ready for testing labels Jul 13, 2023

m2-community-project bot added Progress: extended testing and removed Progress: testing in progress labels Jul 14, 2023

m2-community-project bot added Progress: accept and removed Progress: extended testing labels Jul 17, 2023

m2-assistant bot mentioned this pull request Aug 28, 2023

[Issue] Replaced CREATE TEMPORARY TABLE ... LIKEwith different approach #37926

Closed

5 tasks

m2-community-project bot added Progress: ready for testing and removed Progress: accept labels Aug 28, 2023

m2-community-project bot added Progress: accept and removed Progress: ready for testing labels Aug 28, 2023

magento-devops-reposync-svc merged commit e947b85 into magento:2.4-develop Aug 31, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Replaced `CREATE TEMPORARY TABLE ... LIKE`with different approach #36022

Replaced `CREATE TEMPORARY TABLE ... LIKE`with different approach #36022

Uh oh!

michalbiarda commented Aug 25, 2022 •

edited by m2-assistant bot

Loading

m2-assistant bot commented Aug 25, 2022

michalbiarda commented Aug 25, 2022

magento-deployment-service bot commented Aug 25, 2022

magento-deployment-service bot commented Aug 25, 2022

michalbiarda commented Aug 25, 2022

magento-automated-testing bot commented Aug 25, 2022

kandy Aug 25, 2022

michalbiarda Aug 25, 2022

michalbiarda commented Aug 25, 2022

michalbiarda commented Aug 26, 2022

magento-automated-testing bot commented Aug 26, 2022

engcom-Hotel commented Jun 8, 2023

magento-automated-testing bot commented Jun 8, 2023

engcom-Hotel commented Jun 9, 2023

michalbiarda commented Jun 9, 2023

engcom-Lima commented Jun 9, 2023

magento-automated-testing bot commented Jun 9, 2023

engcom-Hotel commented Jul 13, 2023

engcom-Hotel commented Jul 14, 2023

engcom-Echo commented Jul 17, 2023

magento-automated-testing bot commented Jul 17, 2023

engcom-Echo commented Jul 17, 2023

engcom-Charlie commented Aug 25, 2023

magento-automated-testing bot commented Aug 25, 2023

engcom-Hotel commented Aug 28, 2023

engcom-Hotel commented Aug 28, 2023

kennylawrence commented Nov 4, 2024

kennylawrence commented Nov 4, 2024

	$originCreate = $this->fetchPairs(sprintf('SHOW CREATE TABLE %s', $originTable));
	$originCreate = $this->fetchOne("SHOW CREATE TABLE {$originTable}");

Replaced CREATE TEMPORARY TABLE ... LIKEwith different approach #36022

Replaced CREATE TEMPORARY TABLE ... LIKEwith different approach #36022

Uh oh!

Conversation

michalbiarda commented Aug 25, 2022 • edited by m2-assistant bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description (*)

Manual testing scenarios (*)

Contribution checklist (*)

Resolved issues:

m2-assistant bot commented Aug 25, 2022

michalbiarda commented Aug 25, 2022

magento-deployment-service bot commented Aug 25, 2022

magento-deployment-service bot commented Aug 25, 2022

michalbiarda commented Aug 25, 2022

magento-automated-testing bot commented Aug 25, 2022

kandy Aug 25, 2022

Choose a reason for hiding this comment

michalbiarda Aug 25, 2022

Choose a reason for hiding this comment

michalbiarda commented Aug 25, 2022

michalbiarda commented Aug 26, 2022

magento-automated-testing bot commented Aug 26, 2022

engcom-Hotel commented Jun 8, 2023

magento-automated-testing bot commented Jun 8, 2023

engcom-Hotel commented Jun 9, 2023

michalbiarda commented Jun 9, 2023

engcom-Lima commented Jun 9, 2023

magento-automated-testing bot commented Jun 9, 2023

engcom-Hotel commented Jul 13, 2023

engcom-Hotel commented Jul 14, 2023

engcom-Echo commented Jul 17, 2023

magento-automated-testing bot commented Jul 17, 2023

engcom-Echo commented Jul 17, 2023

engcom-Charlie commented Aug 25, 2023

magento-automated-testing bot commented Aug 25, 2023

engcom-Hotel commented Aug 28, 2023

engcom-Hotel commented Aug 28, 2023

kennylawrence commented Nov 4, 2024

kennylawrence commented Nov 4, 2024

Replaced `CREATE TEMPORARY TABLE ... LIKE`with different approach #36022

Replaced `CREATE TEMPORARY TABLE ... LIKE`with different approach #36022

michalbiarda commented Aug 25, 2022 •

edited by m2-assistant bot

Loading