Master to stage

addons/performance-tuning/README.md

@@ -1,8 +1,8 @@
 # Database Cluster Tuning Add-On
 
-The default configurations of the MySQL/MariaDB database clusters are suitable for a limited number of cases only. If you want to achieve the best performance from your cluster, personalized optimizations are needed.
+The default configurations of the MySQL/MariaDB/Percona database clusters are suitable for a limited number of cases only. If you want to achieve the best performance from your cluster, personalized optimizations are needed.
 
-The tuning add-on is designed to optimize the performance of the MySQL/MariaDB Clusters that work with ProxySQL as a database load balancer. It provides automation to change the most crucial MySQL database parameters, ProxySQL configuration files, and variables to boost overall performance. Add-on's UI will make the process quick and convenient for the database administrators.
+The tuning add-on is designed to optimize the performance of the MySQL/MariaDB/Percona Clusters that work with ProxySQL as a database load balancer. It provides automation to change the most crucial database parameters, ProxySQL configuration files, and variables to boost overall performance. Add-on's UI will make the process quick and convenient for the database administrators.
 
 > Take into consideration that database cluster performance is a complex procedure that evaluates multiple factors, and only a few of them can be solved with the add-on. Check the **Performance Tuning** section below for more information.
 
@@ -11,21 +11,21 @@ The tuning add-on is designed to optimize the performance of the MySQL/MariaDB C
 
 There are multiple factors that can affect the database cluster performance. We’ve categorized the most common and impactful tips on performance tuning into two major areas:
 
-### System MySQL Performance Tuning
+### System Performance Tuning
 
 - Balance four main ***hardware resources***:
  - **Storage**. Take a moment to evaluate your storage. If you’re using traditional hard disk drives (HDD), you can upgrade to solid-state drives (SSD) for significant performance improvement.
  - **Processor**. Processors are usually considered the measure of how fast your system is.
- - **Memory**. Memory represents the total amount of RAM in your MySQL database storage server.
+ - **Memory**. Memory represents the total amount of RAM in your database storage server.
  - **Network**. It is important to monitor network traffic to ensure you have sufficient infrastructure to handle the load.
 - Use ***InnoDB*** instead of *MyISAM*. It offers better performance with large volumes of data and a higher writing speed with rollbacks and commits.
-- Use the ***latest version*** of MySQL/MariaDB. As the stacks are constantly developed, new optimization, performance, and security features are released regularly.
+- Use the ***latest version*** of MySQL/MariaDB/Percona. As the stacks are constantly developed, new optimization, performance, and security features are released regularly.
 
 ### Software Performance Tuning
 
 SQL performance tuning is the process of maximizing query speeds on a relational database. The task usually involves multiple tools and techniques, like:
 
-- Tweaking the MySQL and ProxySQL configuration files.
+- Tweaking the database configuration files.
 - Writing more efficient database queries.
 - Structuring the database to retrieve data more efficiently.
 
@@ -41,35 +41,35 @@ In general, the software performance optimization tips are:
 - Use ORDER BY appropriately.
 - JOIN, WHERE, UNION, DISTINCT. Try to use an inner join whenever possible.
 - Use the EXPLAIN function.
-- **MySQL Server Configuration**. It involves changes to the */etc/mysql/my.cnf* file and MySQL variables.
-- **ProxySQL Server Configuration**. It involves changes to Global Variables and MySQL Servers variables.
+- **Database Server Configuration**. It involves changes to the */etc/mysql/my.cnf* file and database variables.
+- **ProxySQL Server Configuration**. It involves changes to Global Variables and database server variables.
 
 Only the two last points can be adjusted by the add-on to increase database cluster performance. Other tips are dependent on clients' actions on database and query optimization.
 
 
 ## Add-On Installation
 
-The add-on can be installed on top of the [MySQL/MariaDB cluster](https://www.virtuozzo.com/application-platform-docs/auto-clustering/#mysql) with ProxySQL. The appropriate environment should be already installed on your Virtuozzo Application Platform account.
+The add-on can be installed on top of the [MySQL/MariaDB/Percona cluster](https://www.virtuozzo.com/application-platform-docs/db-auto-clustering/) with ProxySQL. The appropriate environment should be already installed on your Virtuozzo Application Platform account.
 
 1\. [Import](https://www.virtuozzo.com/application-platform-docs/environment-import/) the add-on using a link to the ***manifest*** file in this repository:
 
 ```
-https://github.com/jelastic-jps/mysql-cluster/blob/JE-66025/addons/performance-tuning/manifest.yml
+https://github.com/jelastic-jps/mysql-cluster/blob/master/addons/performance-tuning/manifest.yml
 ```
 
-![import add-on](images/01-import-addon.png)
+![import tuning add-on](images/01-import-tuning-addon.png)
 
 Click **Import** to proceed.
 
 2\. Within the opened window, choose the target environment and select the ProxySQL nodes layer.
 
-![install add-on](images/02-install-addon.png)
+![install tuning add-on](images/02-install-tuning-addon.png)
 
 Click **Install** to add the add-on.
 
 3\. Once completed, you can find the add-on under the corresponding tab for the ProxySQL layer.
 
-![change settings add-on](images/03-change-settings-addon.png)
+![change settings tuning add-on](images/03-change-settings-tuning-addon.png)
 
 4\. Click the **Change Settings** button and provide the following parameters:
 

addons/promote-new-primary/README.md

@@ -1,70 +1,91 @@
-# MySQL/MariaDB Primary-Secondary Auto-Failover Add-On
+# Primary-Secondary Auto-Failover Add-On
 
-The add-on is aimed to complement the MySQL/MariaDB Primary-Secondary implementation by handling its main flaw - no automatic failover upon primary failure. As a result, you get the consistency of the Primary-Secondary scheme (i.e. exactly one node to modify data) and simultaneously get auto-failover similar to Primary-Primary.
+The add-on is aimed to complement the MySQL/MariaDB/Percona Primary-Secondary implementation by handling its main flaw - no automatic failover upon primary failure. As a result, you get the consistency of the Primary-Secondary scheme (i.e. exactly one node to modify data) and simultaneously get auto-failover similar to Primary-Primary.
 
 
 ## Add-On Installation
 
-The add-on is designed to work with the Virtuozzo Application Platform auto-clustering implementation of the [MySQL/MariaDB Primary-Secondary](https://www.virtuozzo.com/application-platform-docs/auto-clustering/#mysql) topology only. If you don’t have such an environment yet, you can install one directly via the dashboard:
+The add-on is designed to work with the Virtuozzo Application Platform auto-clustering implementation of the [MySQL/MariaDB/Percona Primary-Secondary](https://www.virtuozzo.com/application-platform-docs/auto-clustering/#templates-with-supported-auto-clustering) topologies only. If you don’t have such an environment yet, you can install one directly via the dashboard:
 
-1\. [Create a new environment](https://www.virtuozzo.com/application-platform-docs/setting-up-environment/). Select the **MySQL** or **MariaDB** node and enable the *Auto-Clustering* feature in the central part of the wizard. Ensure that the ***Primary-Secondary*** scheme is selected and the ***Add ProxySQL*** option is enabled.
+1\. [Create a new environment](https://www.virtuozzo.com/application-platform-docs/setting-up-environment/). Select the **MySQL**, **MariaDB**, or **Percona** node and enable the *Auto-Clustering* feature in the central part of the wizard. Ensure that the ***Primary-Secondary*** scheme is selected and the ***Add ProxySQL*** option is enabled.
 
-![MariaDB Primary-Secondary](images/01-mariadb-primary-secondary.png)
+![create DB cluster](images/01-create-db-cluster.png)
 
 Customize other settings up to your needs and click **Create**.
 
 2\. [Import](https://www.virtuozzo.com/application-platform-docs/environment-import/) the add-on for auto-failover from this repository:
 
 ```
-https://github.com/jelastic-jps/mysql-cluster/blob/JE-66025/addons/promote-new-primary/manifest.yml
+https://github.com/jelastic-jps/mysql-cluster/blob/master/addons/promote-new-primary/addon-with-proxysql.yml
 ```
 
-![import add-on](images/02-import-addon.png)
+![install the promote primary add-on](images/02-install-promote-primary-addon.png)
 
-3\. Select the target environment with the Primary-Secondary database cluster and ProxySQL nodes in the opened installation window. Optionally, you can adjust the timeout for the primary node (no less than *60* seconds), i.e. if not available for the stated period – a new primary will be promoted.
+Provide the following data to set up the add-on:
 
-![install add-on](images/03-install-addon.png)
+- **Primary Idle Time (sec)** - adjust the timeout for the primary node (no less than *30* seconds), i.e. if not available for the stated period – a new primary will be promoted
+- **Endpoint to Primary DB** – add dynamic [endpoint](https://www.virtuozzo.com/application-platform-docs/endpoints/) for direct connection to the Primary database bypassing the ProxySQL node
+- **Environment name** – choose an environment with the Primary-Secondary database cluster
+- **Nodes** – select the ProxySQL layer within the target environment
 
-That’s all! The add-on will be installed in a few minutes.
+That’s all! The add-on will be installed in a minute.
 
 
 ## Add-On Configuration
 
-After installation, you can find the add-on under the corresponding tab for the ProxySQL layer.
+Installed add-on can be found under the corresponding tab for the ProxySQL layer.
 
-![configure add-on](images/04-configure-addon.png)
+![configure promote add-on](images/03-configure-promote-addon.png)
 
-Here, you can adjust the **Primary Idle Time** period (no less than *60* seconds):
+Here, you can adjust the **Auto Failover** settings provided during the installation – change the *Primary Idle Time* period and add/remove *Endpoint to the Primary DB*.
 
-![change timeout](images/05-change-timeout.png)
+![change failover timeout](images/04-change-failover-timeout.png)
 
-If the add-on is not needed anymore, it can be removed with the **Uninstall** button.
+The **Failover** button will simulate the Primary node failure to test failover procedure.
 
-![uninstall add-on](images/06-uninstall-addon.png)
+If the endpoint option was enabled during the installation/configuration, the connection link can be found in the **Endpoints** section of the environment settings.
 
+![primary DB endpoint](images/05-primary-db-endpoint.png)
 
-## Failover Testing
+You can remove the add-on with the **Uninstall** button, if not needed anymore.
 
-You can perform a quick failover testing after the add-on installation.
+![uninstall promote add-on](images/06-uninstall-promote-addon.png)
 
-1\. Connect to the Primary node (marked in the dashboard) via [Web SSH](https://www.virtuozzo.com/application-platform-docs/web-ssh-client/).
 
-![Primary Web SSH](images/07-primary-web-ssh.png)
+## Failover Testing
 
-2\. Stop the node with the following command:
+You can perform a quick failover testing after the add-on installation. The straightforward approach would be to run failover from the add-on's configuration (the **Failover** button).
 
-```
-jem service stop
-```
+![simulate failover cofirmation](images/07-simulate-failover-cofirmation.png)
+
+> Alternatively, you can simulate a problem manually.
+>
+> 1\. Connect to the Primary node (marked in the dashboard) via [Web SSH](https://www.virtuozzo.com/application-platform-docs/web-ssh-client/).
+>
+> ![database Web SSH access](images/08-database-web-ssh-access.png)
+>
+> 2\. Stop the node with the following command:
+>
+> ```
+> jem service stop
+> ```
+>
+> ![stop database service](images/09-stop-database-service.png)
+>
+> 3\. Wait for the timeout period.
+
+In this testing failover scenario, we simulate the Primary node failure. The following actions will be performed automatically to restore the cluster:
 
-![stop Primary](images/08-stop-primary.png)
+- old Primary is excluded from the cluster and removed
+- old Secondary is promoted to a new Primary
+- a new Secondary node is created and added to the cluster
 
-3\. Wait for the timeout period. A new Primary will be promoted, and an additional Secondary database will be added automatically.
+![auto failover result](images/10-auto-failover-result.png)
 
-![new Primary](images/09-new-primary.png)
+You can also check the ProxySQL [logs](https://www.virtuozzo.com/application-platform-docs/view-log-files/) (***jcm.log***) for more information.
 
-4\. You can also check the ProxySQL [logs](https://www.virtuozzo.com/application-platform-docs/view-log-files/) for more information.
+![failover log file](images/11-failover-log.png)
 
-![ProxySQL logs](images/10-proxysql-logs.png)
+As you can see, the Primary status is checked every five seconds. Once it is offline for the whole timeout period (immediately if called via add-on's settings), the new Primary is promoted.
 
-As you can see, the Primary status is checked every five seconds. Once it is offline for the whole timeout period (one minute in our case), the new Primary is promoted.
+If you’ve added an endpoint, it will be automatically reconfigured to connect to the new Primary node. 

addons/promote-new-primary/scripts/promote-master.js

@@ -80,6 +80,10 @@ function promoteNewPrimary() {
  resp = this.addIteration(true);
  if (resp.result != 0) return resp;
  return this.setIsRunningStatus(false);
+ } else {
+ let nodeGroup = this.getAddOnType() ? PROXY : SQLDB;
+
+ return this.cmdByGroup("rm -rf " + TMP_FILE, nodeGroup, 3);
  }
 
  return { result: 0 }
@@ -636,7 +640,7 @@ function promoteNewPrimary() {
  }
  }
 
- let resp = api.env.control.ChangeTopology({
+ return api.env.control.ChangeTopology({
  envName: envName,
  session: session,
  env: {
@@ -645,9 +649,6 @@ function promoteNewPrimary() {
  },
  nodes: nodes
  });
- if (resp.result != 0) return resp;
-
- return this.cmdByGroup("rm -rf " + TMP_FILE, SQLDB, 3);
  };
 
  this.removeFailedPrimary = function() {
@@ -670,7 +671,7 @@ function promoteNewPrimary() {
 
  if (nodeGroup == PROXY && !test) {
  let resp = this.checkNodesAvailability(PROXY);
- if (resp.nodeid) {
+ if (resp && resp.nodeid) {
  return this.cmdById(resp.nodeid, command);
  }
  }

addons/recovery/README.md

@@ -45,11 +45,13 @@ Click **Install** and wait for the successful installation. The add-on will appe
 
 The add-on allows doing two actions:
 
-- **Cluster Diagnostic** - with this action, the add-on automatically scans all nodes in the cluster in order to identify whether the nodes are accessible and whether databases are consistent. If during diagnostic the database corruption or even node failure is detected, the add-on will warn you with a respective popup window:
+- **Cluster Diagnostic** - with this action, the add-on automatically scans all nodes in the cluster in order to identify whether the nodes are accessible and whether databases are consistent. If during diagnostic the database corruption or even node failure is detected, the add-on will warn you with a respective pop-up window:
 
 ![diagnostic failure](images/04-diagnostic-failure.png)
 
-- **Cluster Recovery** - once any failure has been detected, you can either try automatic database recovery by pressing the **Cluster Recovery** button or perform manual database recovery by following the link to the recovery guide. The best practice is to use the automatic recovery scenario.
+> **Tip:** Cluster diagnostic is triggered automatically before the **environment stop** and **cloudlets change** operations. The respective actions will proceed only if the database integrity is verified (to avoid additional damage to the cluster).
+
+- **Cluster Recovery** - if any failure has been detected, you can either try automatic database recovery by pressing the **Cluster Recovery** button or perform manual database recovery by following the link to the recovery guide. The best practice is to use the automatic recovery scenario.
 
 To perform automatic recovery, provide database user credentials either you got upon database cluster installation or the credentials of another privileged user you created.
 

addons/recovery/manifest.yml

@@ -60,7 +60,6 @@ onAfterScaleOut[sqldb]: generate-share-keys
 
 onAfterRedeployContainer[sqldb]: generate-share-keys
 onBeforeStop: eventDiagnostic
-onBeforeSetCloudletCount: eventDiagnostic
 
 onAfterClone:
  install: ${baseUrl}/manifest.yml?_r=${fn.random}
@@ -101,6 +100,11 @@ responses:
  Please check the **/var/log/db_recovery.log** log file for details. Click the "Cluster Recovery" button in the add-on's tab for automatic recovery.
  Follow the [Manual Recovery](https://github.com/jelastic-jps/mysql-cluster/blob/master/addons/recovery/docs/ManualRecoveryGuide.md) guide to recover the cluster manually.
 
+ 96:
+ type: warning
+ message: |
+ mysql or mariadb command not found. 
+
 actions:
  clusterValidate:
  - script: |