Frรฉdรฉric Descamps MySQL Community Manager May 2021 MySQL Shell for DBAs Many questions... one answer !
Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purpose only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied up in making purchasing decisions. The development, release, timing and pricing of any features or functionality described for Oracleยดs product may change and remains at the sole discretion of Oracle Corporation. Copyright @ 2021 Oracle and/or its affiliates. 2
Who am I ? about.me/lefred Copyright @ 2021 Oracle and/or its affiliates. 3
Frรฉdรฉric Descamps @lefred MySQL Evangelist Managing MySQL since 3.20 devops believer living in Belgium h ps://lefred.be Copyright @ 2021 Oracle and/or its affiliates. 4
MySQL Shell a new tool Copyright @ 2021 Oracle and/or its affiliates. 5
MySQL Shell The MySQL Shell is an interactive Javascript, Python, or SQL interface supporting development and administration for the MySQL Server and is a component of the MySQL Server. You can use the MySQL Shell to perform data queries and updates as well as various administration operations. Copyright @ 2021 Oracle and/or its affiliates. 6
MySQL Shell (2) The MySQL Shell provides: Copyright @ 2021 Oracle and/or its affiliates. 7
MySQL Shell (2) The MySQL Shell provides: Both Interactive and Batch operations Copyright @ 2021 Oracle and/or its affiliates. 7
MySQL Shell (2) The MySQL Shell provides: Both Interactive and Batch operations Document and Relational Models Copyright @ 2021 Oracle and/or its affiliates. 7
MySQL Shell (2) The MySQL Shell provides: Both Interactive and Batch operations Document and Relational Models CRUD Document and Relational APIs via scripting Copyright @ 2021 Oracle and/or its affiliates. 7
MySQL Shell (2) The MySQL Shell provides: Both Interactive and Batch operations Document and Relational Models CRUD Document and Relational APIs via scripting Traditional Table, JSON, Tab Separated output results formats Copyright @ 2021 Oracle and/or its affiliates. 7
MySQL Shell (2) The MySQL Shell provides: Both Interactive and Batch operations Document and Relational Models CRUD Document and Relational APIs via scripting Traditional Table, JSON, Tab Separated output results formats MySQL Standard and X Protocols Copyright @ 2021 Oracle and/or its affiliates. 7
MySQL Shell (2) The MySQL Shell provides: Both Interactive and Batch operations Document and Relational Models CRUD Document and Relational APIs via scripting Traditional Table, JSON, Tab Separated output results formats MySQL Standard and X Protocols and more... Copyright @ 2021 Oracle and/or its affiliates. 7
MySQL Shell Overview PY JS SQL > MySQL 8.0 Upgrade Checker Auto Completion & Command History Output Formats (table, json, tabs) Prompt Themes Batch Execution 5.7 8.0 JS Document Store X dev API SQL CLI importJSON InnoDB Cluster JS JS parallel importTABLE data dump & load InnoDB ReplicaSet Copyright @ 2021 Oracle and/or its affiliates. 8
Starting with MySQL Shell setup your environment Copyright @ 2021 Oracle and/or its affiliates. 9
Install MySQL Shell the package's name to install is mysql-shell you always need to use the latest available even with a lower MySQL Server version latest version if 8.0.25, use it even with MySQL 5.7.x Shell Copyright @ 2021 Oracle and/or its affiliates. 10
Start a beautiful Shell The default prompts is basic, we provide several di erent prompt examples. # cp /usr/share/mysqlsh/prompt/prompt_256pl+aw.json ~/.mysqlsh/prompt.json Then start the mysqlsh: Copyright @ 2021 Oracle and/or its affiliates. 11
Some con guration tips These are some se ings I usually use: mysql-js> shell.options.setPersist('history.autoSave', 1) mysql-js> shell.options.setPersist('history.maxSize', 5000) You can switch to all 3 di erent modes using: py for Python js for Javascript sql for SQL If you have a prefered mode, you can specify it like this: mysql-js> shell.options.setPersist('defaultMode', 'sql') Copyright @ 2021 Oracle and/or its affiliates. 12
What does a MySQL DBA do? De ne the tasks Copyright @ 2021 Oracle and/or its affiliates. 13
Top 5 MySQL DBA Tasks . Deploy the architectures (or at least explains what/how). . Prepare upgrades . Dump & Load data . Manage users . Understand the workload Copyright @ 2021 Oracle and/or its affiliates. 14
Deploying Architectures MySQL Shell Admin API Copyright @ 2021 Oracle and/or its affiliates. 15
MySQL Shell Admin API The AdminAPI is an API that enables con guring and managing InnoDB clusters and replicasets, among other things. The AdminAPI is available via the dba global object of MySQL Shell Copyright @ 2021 Oracle and/or its affiliates. 16
MySQL Shell Admin API The AdminAPI is an API that enables con guring and managing InnoDB clusters and replicasets, among other things. The AdminAPI is available via the dba global object of MySQL Shell This provides a very easy and secure way to deploy MySQL Architectures for the DBA. It includes con guration, checks and even data provisioning ! Copyright @ 2021 Oracle and/or its affiliates. 16
MySQL Shell Admin API (2) As like most of the MySQL Shell capabilities all the Admin API calls can be done interactively or directly from the command line or a script. This allows automation with a con guration management tool for example. Copyright @ 2021 Oracle and/or its affiliates. 17
The new MySQL Operator for K8s also uses MySQL Shell in the backend. MySQL Shell Admin API (2) As like most of the MySQL Shell capabilities all the Admin API calls can be done interactively or directly from the command line or a script. This allows automation with a con guration management tool for example. Copyright @ 2021 Oracle and/or its affiliates. 17
MySQL Shell Admin API - examples Copyright @ 2021 Oracle and/or its affiliates. 18
MySQL InnoDB Cluster MySQL InnoDB ReplicaSet js> c admin@mysql1 js> cluster = dba.createCluster('cluster') js> c admin@mysql1 js> rs = dba.createReplicaSet('replicaset') MySQL Shell Admin API - examples Copyright @ 2021 Oracle and/or its affiliates. 18
MySQL InnoDB Cluster MySQL InnoDB ReplicaSet js> c admin@mysql1 js> cluster = dba.createCluster('cluster') js> c admin@mysql1 js> rs = dba.createReplicaSet('replicaset') Con gure server to add later js> dba.con gureInstance('admin@mysql2') js> dba.con gureReplicaSetInstance('admin@mysql2') MySQL Shell Admin API - examples Copyright @ 2021 Oracle and/or its affiliates. 18
MySQL InnoDB Cluster MySQL InnoDB ReplicaSet js> c admin@mysql1 js> cluster = dba.createCluster('cluster') js> c admin@mysql1 js> rs = dba.createReplicaSet('replicaset') Con gure server to add later js> dba.con gureInstance('admin@mysql2') js> dba.con gureReplicaSetInstance('admin@mysql2') Add server to the Cluster js> cluster.addInstance('admin@mysql2') js> rs.addInstance('admin@mysql2') MySQL Shell Admin API - examples Copyright @ 2021 Oracle and/or its affiliates. 18
MySQL InnoDB Cluster MySQL InnoDB ReplicaSet js> c admin@mysql1 js> cluster = dba.createCluster('cluster') js> c admin@mysql1 js> rs = dba.createReplicaSet('replicaset') Con gure server to add later js> dba.con gureInstance('admin@mysql2') js> dba.con gureReplicaSetInstance('admin@mysql2') Add server to the Cluster js> cluster.addInstance('admin@mysql2') js> rs.addInstance('admin@mysql2') Bootstrap MySQL Router $ sudo mysqlrouter --user=mysqlrouter --bootstrap $ sudo systemctl start mysqlrouter $ sudo mysqlrouter --user=mysqlrouter --bootstrap $ sudo systemctl start mysqlrouter MySQL Shell Admin API - examples Copyright @ 2021 Oracle and/or its affiliates. 18
MySQL Shell Admin API (3) The Admin API also provides methods to manage your MySQL Architecture. Copyright @ 2021 Oracle and/or its affiliates. 19
addInstance checkInstanceState describe dissolve forceQuorumUsingPartitionOf listRouters rejoinInstance removeInstance removeRouterMetadata resetRecoveryAccountsPassword setInstanceOption setOption setPrimaryInstance setupAdminAccount setupRouterAccount status switchToMultiPrimaryMode switchToSinglePrimaryMode MySQL Shell Admin API (3) The Admin API also provides methods to manage your MySQL Architecture. Copyright @ 2021 Oracle and/or its affiliates. 19
Prepare Upgrades let's move to 8.0 ! Copyright @ 2021 Oracle and/or its affiliates. 20
Upgrade to MySQL 8.0 MySQL Shell helps the DBA to validate the upgrade to MySQL 8.0. Copyright @ 2021 Oracle and/or its affiliates. 21
Upgrade to MySQL 8.0 MySQL Shell helps the DBA to validate the upgrade to MySQL 8.0. With util.checkForServerUpgrade, MySQL Shell can perform a series of tests on a speci ed MySQL sever to check is the upgrade process will succeed or if requires rst some changes. Copyright @ 2021 Oracle and/or its affiliates. 21
Upgrade to MySQL 8.0 MySQL Shell helps the DBA to validate the upgrade to MySQL 8.0. With util.checkForServerUpgrade, MySQL Shell can perform a series of tests on a speci ed MySQL sever to check is the upgrade process will succeed or if requires rst some changes. Will check your MySQL 5.7 or 8.0 installation readiness for upgrade to latest version Check for legacy issues Run the tool before upgrading! Always use latest version of MySQL Shell Copyright @ 2021 Oracle and/or its affiliates. 21
usage of old temporal type usage of new reserved keywords usage of utf8mb3 charset con icting name in mysql schema partitions with engine not supporting native partitioning foreign key constraint names longer than 64 characters usage of obsolete MAXDB SQL mode ag usage of orther obsolete SQL mode ags ENUM/SET column de nitions containing elements longer than 255 characters usage of partitioned tables in shared tablespaces MySQL Shell Upgrade Checker Utility Upgrade Checker performs the following checks: Copyright @ 2021 Oracle and/or its affiliates. 22
circular directory references in tablespace data le paths usage of removed functions usage of removed GROUP BY ASC/DESC syntax removed system variables for error logging to the system log con guration removed system variables system variables with new default values schema inconsistencies resulting from le removal or corruption table names containing 'FTS' issues reported by check table x for upgrade command MySQL Shell Upgrade Checker Utility (2) Copyright @ 2021 Oracle and/or its affiliates. 23
MySQL Upgrade Checker: how to use it? From the system shell: $ mysqlsh -- util checkForServerUpgrade user@localhost:3306 --target-version=8.0.24 --output-format=JSON --con g-path=/etc/mysql/my.cnf Interactively from MySQL Shell: mysqlsh> util.checkForServerUpgrade('user@example.com:3306', {"password":"password", "targetVersion":"8.0.24", "con gPath":"/etc/my.cnf"}) mysqlsh> util.checkForServerUpgrade() Copyright @ 2021 Oracle and/or its affiliates. 24
MySQL Upgrade Checker - output example 16) System variables with new default values Warning: Following system variables that are not de ned in your con guration le will have new default values. Please review if you rely on their current values and if so de ne them before performing upgrade. More information: https://mysqlserverteam.com/new-defaults-in-mysql-8-0/ back_log - default value will change character_set_server - default value will change from latin1 to utf8mb4 collation_server - default value will change from latin1_swedish_ci to utf8mb4_0900_ai_ci event_scheduler - default value will change from OFF to ON explicit_defaults_for_timestamp - default value will change from OFF to ON innodb_autoinc_lock_mode - default value will change from 1 (consecutive) to 2 (interleaved) innodb_ ush_method - default value will change from NULL to fsync (Unix), unbuffered (Windows) innodb_ ush_neighbors - default value will change from 1 (enable) to 0 (disable) innodb_max_dirty_pages_pct - default value will change from 75 (%) 90 (%) innodb_max_dirty_pages_pct_lwm - default value will change from_0 (%) to 10 (%) innodb_undo_log_truncate - default value will change from OFF to ON ... Copyright @ 2021 Oracle and/or its affiliates. 25
MySQL Upgrade Checker - output example (2) 17) Zero Date, Datetime, and Timestamp values Warning: By default zero date/datetime/timestamp values are no longer allowed in MySQL, as of 5.7.8 NO_ZERO_IN_DATE and NO_ZERO_DATE are included in SQL_MODE by default. These modes should be used with strict mode as they will be merged with strict mode in a future release. If you do not include these modes in your SQL_MODE setting, you are able to insert date/datetime/timestamp values that contain zeros. It is strongly advised to replace zero values with valid ones, as they may not work correctly in the future. More information: https://lefred.be/content/mysql-8-0-and-wrong-dates/ sbtest_tpcc.orders2.o_entry_d - column has zero default value: 0000-00-00 00:00:00 Copyright @ 2021 Oracle and/or its affiliates. 26
Dump & Load Data Logical Dumps and Data imports Copyright @ 2021 Oracle and/or its affiliates. 27
Logical Dumps Another aspect of the DBAs daily tasks is to deal with logical dumps, for backup, for migrations, to reset data in dev environments ... Copyright @ 2021 Oracle and/or its affiliates. 28
Logical Dumps Another aspect of the DBAs daily tasks is to deal with logical dumps, for backup, for migrations, to reset data in dev environments ... MySQL Shell provides and amazing tool that relegates all its competitors far behind ! Copyright @ 2021 Oracle and/or its affiliates. 28
Logical Dumps Another aspect of the DBAs daily tasks is to deal with logical dumps, for backup, for migrations, to reset data in dev environments ... MySQL Shell provides and amazing tool that relegates all its competitors far behind ! Bye bye mysqldump, mysqlpump, mydumper, ... Copyright @ 2021 Oracle and/or its affiliates. 28
Logical Dumps Another aspect of the DBAs daily tasks is to deal with logical dumps, for backup, for migrations, to reset data in dev environments ... MySQL Shell provides and amazing tool that relegates all its competitors far behind ! Bye bye mysqldump, mysqlpump, mydumper, ... Welcome MySQL Shell Load & Dump !! Copyright @ 2021 Oracle and/or its affiliates. 28
Logical Dumps (2) 3 types of dumps are available and are accessible via those methods: dumpInstance() dumpSchemas() dumpTables() Copyright @ 2021 Oracle and/or its affiliates. 29
dump to remote storage (like Object Store) progress state tracking check for compatibility with MDS resume and reset capability add rate limitation use compression on the y changes: create invisible PKs force InnoDB ignore missing PKs skip invalid accounts strip de ners strip restricted grants strip tablespaces Logical Dumps (3) Dumps are taken in parallel but has many other bene ts like : Copyright @ 2021 Oracle and/or its affiliates. 30
run analyze tables after load (can be limited to those having histograms) the character set can be overridden invisible PKs creation defer the creation of secondary indexes (can also be limited to FT) just load the scheme's de nition or the data resume and reset capability show progress skip binary logs update or override GTIDs Logical Load The previous dumps can also be loaded in parallel and many parameters are supported to provide the best experience. Let's see some of them: Copyright @ 2021 Oracle and/or its affiliates. 31
Logical Load (2) Take a look ๐Ÿ‘€at this output: Copyright @ 2021 Oracle and/or its affiliates. 32
Logical Load (2) Take a look ๐Ÿ‘€at this output: Copyright @ 2021 Oracle and/or its affiliates. 32
Logical Load (2) Take a look ๐Ÿ‘€at this output: MySQL Shell is also the easiest and fastest way to migrate to MySQL Database Service on OCI CLOUD Copyright @ 2021 Oracle and/or its affiliates. 32
Users & Privileges MySQL Accounts and Grants Copyright @ 2021 Oracle and/or its affiliates. 33
Users & Privileges An other important task of MySQL DBAs is to manage who's accessing the database. Copyright @ 2021 Oracle and/or its affiliates. 34
Users & Privileges An other important task of MySQL DBAs is to manage who's accessing the database. Security is very important for MySQL and 8.0 provides many features related to user accounts. We also introduced more granular privileges. Copyright @ 2021 Oracle and/or its affiliates. 34
ROLES new more secure authentication method caching_sha2_password comments & json a ributes random password more options regarding old passwords: history reuse interval current password requirement lock time, ... Users & Privileges An other important task of MySQL DBAs is to manage who's accessing the database. Security is very important for MySQL and 8.0 provides many features related to user accounts. We also introduced more granular privileges. MySQL 8.0 introduced: Copyright @ 2021 Oracle and/or its affiliates. 34
Extending MySQL Shell For repetitive or complex tasks, it's also possible to extend MySQL Shell with plugins. Copyright @ 2021 Oracle and/or its affiliates. 35
Extending MySQL Shell For repetitive or complex tasks, it's also possible to extend MySQL Shell with plugins. User account management is very good candidate for plugins. Copyright @ 2021 Oracle and/or its affiliates. 35
Extending MySQL Shell For repetitive or complex tasks, it's also possible to extend MySQL Shell with plugins. User account management is very good candidate for plugins. Some words on MySQL Shell Plugins: can be wri en in Python or JavaScript are loaded when starting MySQL Shell can use any local modules too Copyright @ 2021 Oracle and/or its affiliates. 35
Extending MySQL Shell For repetitive or complex tasks, it's also possible to extend MySQL Shell with plugins. User account management is very good candidate for plugins. Some words on MySQL Shell Plugins: can be wri en in Python or JavaScript are loaded when starting MySQL Shell can use any local modules too You can nd multiple plugins to test and extend on: h ps://github.com/lefred/mysqlshell-plugins Copyright @ 2021 Oracle and/or its affiliates. 35
Users & Privileges - plugins examples Dumping all users and grants: Copyright @ 2021 Oracle and/or its affiliates. 36
Users & Privileges - plugins examples Copy users from a server to another: Copyright @ 2021 Oracle and/or its affiliates. 37
Users & Privileges - plugins examples Create a user wizard Copyright @ 2021 Oracle and/or its affiliates. 38
Users & Privileges - plugins examples Create a user wizard Copyright @ 2021 Oracle and/or its affiliates. 38
Users & Privileges - plugins examples Authentication methods and passwords to expire: Copyright @ 2021 Oracle and/or its affiliates. 39
Understand the Workload What's happening ? Copyright @ 2021 Oracle and/or its affiliates. 40
Observability in MySQL 8.0 Information_Schema, Performance_Schema and sys schema are in constant evolution and many new metrics appeared in MySQL 8.0. Copyright @ 2021 Oracle and/or its affiliates. 41
Observability in MySQL 8.0 Information_Schema, Performance_Schema and sys schema are in constant evolution and many new metrics appeared in MySQL 8.0. Unless you deal with those schemes everyday, it might be complicated to easily nd the information you are looking for. MySQL Shell can again be helpful here. Copyright @ 2021 Oracle and/or its affiliates. 41
Observability in MySQL 8.0 Information_Schema, Performance_Schema and sys schema are in constant evolution and many new metrics appeared in MySQL 8.0. Unless you deal with those schemes everyday, it might be complicated to easily nd the information you are looking for. MySQL Shell can again be helpful here. There are already several plugins that provide useful information to be er understand the workload of the database. Copyright @ 2021 Oracle and/or its affiliates. 41
Workload examples Copyright @ 2021 Oracle and/or its affiliates. 42
Workload examples Copyright @ 2021 Oracle and/or its affiliates. 42
Workload examples Copyright @ 2021 Oracle and/or its affiliates. 43
Workload examples More info and examples: h ps://github.com/lefred/mysqlshell-plugins/wiki Copyright @ 2021 Oracle and/or its affiliates. 43
Fight the slow and ine cient queries ! JS > check.getFullTableScanQuery() *************************** 1. row *************************** schema_name: test sum_rows_examined: 21 avg_rows_call: 21.0000 tot_lat: 98.20 ms exec_count: 1 latency_per_call: 98.20 ms rst_seen: 2020-07-31 11:51:16.423903 last_seen: 2020-07-31 11:51:16.423903 query_sample_text: select * from users Copyright @ 2021 Oracle and/or its affiliates. 44
Fight the slow and ine cient queries ! (2) Do you want to have EXPLAIN output? (y/N) y *************************** 1. row *************************** id: 1 select_type: SIMPLE table: users partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 18 ltered: 100 Extra: NULL Copyright @ 2021 Oracle and/or its affiliates. 45
Fight the slow and ine cient queries ! (3) Do you want to have EXPLAIN in JSON format output? (y/N) y *************************** 1. row *************************** EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "2.80" }, "table": { "table_name": "users", "access_type": "ALL", "rows_examined_per_scan": 18, "rows_produced_per_join": 18, " ltered": "100.00", "cost_info": { "read_cost": "1.00", "eval_cost": "1.80", "pre x_cost": "2.80", "data_read_per_join": "576" }, "used_columns": [ "user_id", "user_name", "salary" ] } } } Copyright @ 2021 Oracle and/or its affiliates. 46
Fight the slow and ine cient queries ! (4) Do you want to have EXPLAIN in TREE format output? (y/N) y *************************** 1. row *************************** EXPLAIN: -> Table scan on users (cost=2.80 rows=18) Do you want to have EXPLAIN ANALYZE output? (y/N) y *************************** 1. row *************************** EXPLAIN: -> Table scan on users (cost=2.80 rows=18) (actual time=0.041..0.067 rows=21 loops=1) Copyright @ 2021 Oracle and/or its affiliates. 47
Do you remember the old SHOW PROFILE ? You can get similar information using the pro ling plugin for MySQL Shell MySQL Shell - Pro ler Copyright @ 2021 Oracle and/or its affiliates. 48
And also for another foreground thread: MySQL Shell - Pro ler (2) Copyright @ 2021 Oracle and/or its affiliates. 49
Query Execution Plan Another very important part of the MySQL DBA is to control the QEP of the queries. There is a plugin that allows you to run the QEP for a query, save it and compare them. It supports di erent MySQL 8.0 versions. Copyright @ 2021 Oracle and/or its affiliates. 50
Query Execution Plan (2) Copyright @ 2021 Oracle and/or its affiliates. 51
Use and contribute to MySQL Shell Plugins Get the code from h ps://github.com/lefred/mysqlshell-plugins and Pull Requests are welcome ! Copyright @ 2021 Oracle and/or its affiliates. 52
Something more... more control Copyright @ 2021 Oracle and/or its affiliates. 53
con g.getGlobalNonDefault() con g.getSessionNonDefault() con g.getPersistedVariables() con g.getVariableInfo() Who changed what ? Sometimes the DBA needs to ght with con guration changes... This information related to variables is available from the con g MySQL Shell Plugin: Copyright @ 2021 Oracle and/or its affiliates. 54
con g.getGlobalNonDefault() con g.getSessionNonDefault() con g.getPersistedVariables() con g.getVariableInfo() Who changed what ? Sometimes the DBA needs to ght with con guration changes... This information related to variables is available from the con g MySQL Shell Plugin: Copyright @ 2021 Oracle and/or its affiliates. 54
Con guring Variables - MySQL Shell (2) Copyright @ 2021 Oracle and/or its affiliates. 55
Con guring Variables - MySQL Shell (3) Copyright @ 2021 Oracle and/or its affiliates. 56
Thank you ! Copyright @ 2021 Oracle and/or its affiliates. 57
Questions ? Copyright @ 2021 Oracle and/or its affiliates. 58

MySQL Shell for DBAs

  • 1.
    Frรฉdรฉric Descamps MySQL CommunityManager May 2021 MySQL Shell for DBAs Many questions... one answer !
  • 2.
    Safe Harbor Statement Thefollowing is intended to outline our general product direction. It is intended for information purpose only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied up in making purchasing decisions. The development, release, timing and pricing of any features or functionality described for Oracleยดs product may change and remains at the sole discretion of Oracle Corporation. Copyright @ 2021 Oracle and/or its affiliates. 2
  • 3.
    Who am I? about.me/lefred Copyright @ 2021 Oracle and/or its affiliates. 3
  • 4.
    Frรฉdรฉric Descamps @lefred MySQL Evangelist ManagingMySQL since 3.20 devops believer living in Belgium h ps://lefred.be Copyright @ 2021 Oracle and/or its affiliates. 4
  • 5.
    MySQL Shell a newtool Copyright @ 2021 Oracle and/or its affiliates. 5
  • 6.
    MySQL Shell The MySQLShell is an interactive Javascript, Python, or SQL interface supporting development and administration for the MySQL Server and is a component of the MySQL Server. You can use the MySQL Shell to perform data queries and updates as well as various administration operations. Copyright @ 2021 Oracle and/or its affiliates. 6
  • 7.
    MySQL Shell (2) TheMySQL Shell provides: Copyright @ 2021 Oracle and/or its affiliates. 7
  • 8.
    MySQL Shell (2) TheMySQL Shell provides: Both Interactive and Batch operations Copyright @ 2021 Oracle and/or its affiliates. 7
  • 9.
    MySQL Shell (2) TheMySQL Shell provides: Both Interactive and Batch operations Document and Relational Models Copyright @ 2021 Oracle and/or its affiliates. 7
  • 10.
    MySQL Shell (2) TheMySQL Shell provides: Both Interactive and Batch operations Document and Relational Models CRUD Document and Relational APIs via scripting Copyright @ 2021 Oracle and/or its affiliates. 7
  • 11.
    MySQL Shell (2) TheMySQL Shell provides: Both Interactive and Batch operations Document and Relational Models CRUD Document and Relational APIs via scripting Traditional Table, JSON, Tab Separated output results formats Copyright @ 2021 Oracle and/or its affiliates. 7
  • 12.
    MySQL Shell (2) TheMySQL Shell provides: Both Interactive and Batch operations Document and Relational Models CRUD Document and Relational APIs via scripting Traditional Table, JSON, Tab Separated output results formats MySQL Standard and X Protocols Copyright @ 2021 Oracle and/or its affiliates. 7
  • 13.
    MySQL Shell (2) TheMySQL Shell provides: Both Interactive and Batch operations Document and Relational Models CRUD Document and Relational APIs via scripting Traditional Table, JSON, Tab Separated output results formats MySQL Standard and X Protocols and more... Copyright @ 2021 Oracle and/or its affiliates. 7
  • 14.
    MySQL Shell Overview PY JS SQL > MySQL8.0 Upgrade Checker Auto Completion & Command History Output Formats (table, json, tabs) Prompt Themes Batch Execution 5.7 8.0 JS Document Store X dev API SQL CLI importJSON InnoDB Cluster JS JS parallel importTABLE data dump & load InnoDB ReplicaSet Copyright @ 2021 Oracle and/or its affiliates. 8
  • 15.
    Starting with MySQLShell setup your environment Copyright @ 2021 Oracle and/or its affiliates. 9
  • 16.
    Install MySQL Shell thepackage's name to install is mysql-shell you always need to use the latest available even with a lower MySQL Server version latest version if 8.0.25, use it even with MySQL 5.7.x Shell Copyright @ 2021 Oracle and/or its affiliates. 10
  • 17.
    Start a beautifulShell The default prompts is basic, we provide several di erent prompt examples. # cp /usr/share/mysqlsh/prompt/prompt_256pl+aw.json ~/.mysqlsh/prompt.json Then start the mysqlsh: Copyright @ 2021 Oracle and/or its affiliates. 11
  • 18.
    Some con gurationtips These are some se ings I usually use: mysql-js> shell.options.setPersist('history.autoSave', 1) mysql-js> shell.options.setPersist('history.maxSize', 5000) You can switch to all 3 di erent modes using: py for Python js for Javascript sql for SQL If you have a prefered mode, you can specify it like this: mysql-js> shell.options.setPersist('defaultMode', 'sql') Copyright @ 2021 Oracle and/or its affiliates. 12
  • 19.
    What does aMySQL DBA do? De ne the tasks Copyright @ 2021 Oracle and/or its affiliates. 13
  • 20.
    Top 5 MySQLDBA Tasks . Deploy the architectures (or at least explains what/how). . Prepare upgrades . Dump & Load data . Manage users . Understand the workload Copyright @ 2021 Oracle and/or its affiliates. 14
  • 21.
    Deploying Architectures MySQL ShellAdmin API Copyright @ 2021 Oracle and/or its affiliates. 15
  • 22.
    MySQL Shell AdminAPI The AdminAPI is an API that enables con guring and managing InnoDB clusters and replicasets, among other things. The AdminAPI is available via the dba global object of MySQL Shell Copyright @ 2021 Oracle and/or its affiliates. 16
  • 23.
    MySQL Shell AdminAPI The AdminAPI is an API that enables con guring and managing InnoDB clusters and replicasets, among other things. The AdminAPI is available via the dba global object of MySQL Shell This provides a very easy and secure way to deploy MySQL Architectures for the DBA. It includes con guration, checks and even data provisioning ! Copyright @ 2021 Oracle and/or its affiliates. 16
  • 24.
    MySQL Shell AdminAPI (2) As like most of the MySQL Shell capabilities all the Admin API calls can be done interactively or directly from the command line or a script. This allows automation with a con guration management tool for example. Copyright @ 2021 Oracle and/or its affiliates. 17
  • 25.
    The new MySQLOperator for K8s also uses MySQL Shell in the backend. MySQL Shell Admin API (2) As like most of the MySQL Shell capabilities all the Admin API calls can be done interactively or directly from the command line or a script. This allows automation with a con guration management tool for example. Copyright @ 2021 Oracle and/or its affiliates. 17
  • 26.
    MySQL Shell AdminAPI - examples Copyright @ 2021 Oracle and/or its affiliates. 18
  • 27.
    MySQL InnoDB ClusterMySQL InnoDB ReplicaSet js> c admin@mysql1 js> cluster = dba.createCluster('cluster') js> c admin@mysql1 js> rs = dba.createReplicaSet('replicaset') MySQL Shell Admin API - examples Copyright @ 2021 Oracle and/or its affiliates. 18
  • 28.
    MySQL InnoDB ClusterMySQL InnoDB ReplicaSet js> c admin@mysql1 js> cluster = dba.createCluster('cluster') js> c admin@mysql1 js> rs = dba.createReplicaSet('replicaset') Con gure server to add later js> dba.con gureInstance('admin@mysql2') js> dba.con gureReplicaSetInstance('admin@mysql2') MySQL Shell Admin API - examples Copyright @ 2021 Oracle and/or its affiliates. 18
  • 29.
    MySQL InnoDB ClusterMySQL InnoDB ReplicaSet js> c admin@mysql1 js> cluster = dba.createCluster('cluster') js> c admin@mysql1 js> rs = dba.createReplicaSet('replicaset') Con gure server to add later js> dba.con gureInstance('admin@mysql2') js> dba.con gureReplicaSetInstance('admin@mysql2') Add server to the Cluster js> cluster.addInstance('admin@mysql2') js> rs.addInstance('admin@mysql2') MySQL Shell Admin API - examples Copyright @ 2021 Oracle and/or its affiliates. 18
  • 30.
    MySQL InnoDB ClusterMySQL InnoDB ReplicaSet js> c admin@mysql1 js> cluster = dba.createCluster('cluster') js> c admin@mysql1 js> rs = dba.createReplicaSet('replicaset') Con gure server to add later js> dba.con gureInstance('admin@mysql2') js> dba.con gureReplicaSetInstance('admin@mysql2') Add server to the Cluster js> cluster.addInstance('admin@mysql2') js> rs.addInstance('admin@mysql2') Bootstrap MySQL Router $ sudo mysqlrouter --user=mysqlrouter --bootstrap $ sudo systemctl start mysqlrouter $ sudo mysqlrouter --user=mysqlrouter --bootstrap $ sudo systemctl start mysqlrouter MySQL Shell Admin API - examples Copyright @ 2021 Oracle and/or its affiliates. 18
  • 31.
    MySQL Shell AdminAPI (3) The Admin API also provides methods to manage your MySQL Architecture. Copyright @ 2021 Oracle and/or its affiliates. 19
  • 32.
  • 33.
    Prepare Upgrades let's moveto 8.0 ! Copyright @ 2021 Oracle and/or its affiliates. 20
  • 34.
    Upgrade to MySQL8.0 MySQL Shell helps the DBA to validate the upgrade to MySQL 8.0. Copyright @ 2021 Oracle and/or its affiliates. 21
  • 35.
    Upgrade to MySQL8.0 MySQL Shell helps the DBA to validate the upgrade to MySQL 8.0. With util.checkForServerUpgrade, MySQL Shell can perform a series of tests on a speci ed MySQL sever to check is the upgrade process will succeed or if requires rst some changes. Copyright @ 2021 Oracle and/or its affiliates. 21
  • 36.
    Upgrade to MySQL8.0 MySQL Shell helps the DBA to validate the upgrade to MySQL 8.0. With util.checkForServerUpgrade, MySQL Shell can perform a series of tests on a speci ed MySQL sever to check is the upgrade process will succeed or if requires rst some changes. Will check your MySQL 5.7 or 8.0 installation readiness for upgrade to latest version Check for legacy issues Run the tool before upgrading! Always use latest version of MySQL Shell Copyright @ 2021 Oracle and/or its affiliates. 21
  • 37.
    usage of oldtemporal type usage of new reserved keywords usage of utf8mb3 charset con icting name in mysql schema partitions with engine not supporting native partitioning foreign key constraint names longer than 64 characters usage of obsolete MAXDB SQL mode ag usage of orther obsolete SQL mode ags ENUM/SET column de nitions containing elements longer than 255 characters usage of partitioned tables in shared tablespaces MySQL Shell Upgrade Checker Utility Upgrade Checker performs the following checks: Copyright @ 2021 Oracle and/or its affiliates. 22
  • 38.
    circular directory referencesin tablespace data le paths usage of removed functions usage of removed GROUP BY ASC/DESC syntax removed system variables for error logging to the system log con guration removed system variables system variables with new default values schema inconsistencies resulting from le removal or corruption table names containing 'FTS' issues reported by check table x for upgrade command MySQL Shell Upgrade Checker Utility (2) Copyright @ 2021 Oracle and/or its affiliates. 23
  • 39.
    MySQL Upgrade Checker:how to use it? From the system shell: $ mysqlsh -- util checkForServerUpgrade user@localhost:3306 --target-version=8.0.24 --output-format=JSON --con g-path=/etc/mysql/my.cnf Interactively from MySQL Shell: mysqlsh> util.checkForServerUpgrade('user@example.com:3306', {"password":"password", "targetVersion":"8.0.24", "con gPath":"/etc/my.cnf"}) mysqlsh> util.checkForServerUpgrade() Copyright @ 2021 Oracle and/or its affiliates. 24
  • 40.
    MySQL Upgrade Checker- output example 16) System variables with new default values Warning: Following system variables that are not de ned in your con guration le will have new default values. Please review if you rely on their current values and if so de ne them before performing upgrade. More information: https://mysqlserverteam.com/new-defaults-in-mysql-8-0/ back_log - default value will change character_set_server - default value will change from latin1 to utf8mb4 collation_server - default value will change from latin1_swedish_ci to utf8mb4_0900_ai_ci event_scheduler - default value will change from OFF to ON explicit_defaults_for_timestamp - default value will change from OFF to ON innodb_autoinc_lock_mode - default value will change from 1 (consecutive) to 2 (interleaved) innodb_ ush_method - default value will change from NULL to fsync (Unix), unbuffered (Windows) innodb_ ush_neighbors - default value will change from 1 (enable) to 0 (disable) innodb_max_dirty_pages_pct - default value will change from 75 (%) 90 (%) innodb_max_dirty_pages_pct_lwm - default value will change from_0 (%) to 10 (%) innodb_undo_log_truncate - default value will change from OFF to ON ... Copyright @ 2021 Oracle and/or its affiliates. 25
  • 41.
    MySQL Upgrade Checker- output example (2) 17) Zero Date, Datetime, and Timestamp values Warning: By default zero date/datetime/timestamp values are no longer allowed in MySQL, as of 5.7.8 NO_ZERO_IN_DATE and NO_ZERO_DATE are included in SQL_MODE by default. These modes should be used with strict mode as they will be merged with strict mode in a future release. If you do not include these modes in your SQL_MODE setting, you are able to insert date/datetime/timestamp values that contain zeros. It is strongly advised to replace zero values with valid ones, as they may not work correctly in the future. More information: https://lefred.be/content/mysql-8-0-and-wrong-dates/ sbtest_tpcc.orders2.o_entry_d - column has zero default value: 0000-00-00 00:00:00 Copyright @ 2021 Oracle and/or its affiliates. 26
  • 42.
    Dump & LoadData Logical Dumps and Data imports Copyright @ 2021 Oracle and/or its affiliates. 27
  • 43.
    Logical Dumps Another aspectof the DBAs daily tasks is to deal with logical dumps, for backup, for migrations, to reset data in dev environments ... Copyright @ 2021 Oracle and/or its affiliates. 28
  • 44.
    Logical Dumps Another aspectof the DBAs daily tasks is to deal with logical dumps, for backup, for migrations, to reset data in dev environments ... MySQL Shell provides and amazing tool that relegates all its competitors far behind ! Copyright @ 2021 Oracle and/or its affiliates. 28
  • 45.
    Logical Dumps Another aspectof the DBAs daily tasks is to deal with logical dumps, for backup, for migrations, to reset data in dev environments ... MySQL Shell provides and amazing tool that relegates all its competitors far behind ! Bye bye mysqldump, mysqlpump, mydumper, ... Copyright @ 2021 Oracle and/or its affiliates. 28
  • 46.
    Logical Dumps Another aspectof the DBAs daily tasks is to deal with logical dumps, for backup, for migrations, to reset data in dev environments ... MySQL Shell provides and amazing tool that relegates all its competitors far behind ! Bye bye mysqldump, mysqlpump, mydumper, ... Welcome MySQL Shell Load & Dump !! Copyright @ 2021 Oracle and/or its affiliates. 28
  • 47.
    Logical Dumps (2) 3types of dumps are available and are accessible via those methods: dumpInstance() dumpSchemas() dumpTables() Copyright @ 2021 Oracle and/or its affiliates. 29
  • 48.
    dump to remotestorage (like Object Store) progress state tracking check for compatibility with MDS resume and reset capability add rate limitation use compression on the y changes: create invisible PKs force InnoDB ignore missing PKs skip invalid accounts strip de ners strip restricted grants strip tablespaces Logical Dumps (3) Dumps are taken in parallel but has many other bene ts like : Copyright @ 2021 Oracle and/or its affiliates. 30
  • 49.
    run analyze tablesafter load (can be limited to those having histograms) the character set can be overridden invisible PKs creation defer the creation of secondary indexes (can also be limited to FT) just load the scheme's de nition or the data resume and reset capability show progress skip binary logs update or override GTIDs Logical Load The previous dumps can also be loaded in parallel and many parameters are supported to provide the best experience. Let's see some of them: Copyright @ 2021 Oracle and/or its affiliates. 31
  • 50.
    Logical Load (2) Takea look ๐Ÿ‘€at this output: Copyright @ 2021 Oracle and/or its affiliates. 32
  • 51.
    Logical Load (2) Takea look ๐Ÿ‘€at this output: Copyright @ 2021 Oracle and/or its affiliates. 32
  • 52.
    Logical Load (2) Takea look ๐Ÿ‘€at this output: MySQL Shell is also the easiest and fastest way to migrate to MySQL Database Service on OCI CLOUD Copyright @ 2021 Oracle and/or its affiliates. 32
  • 53.
    Users & Privileges MySQLAccounts and Grants Copyright @ 2021 Oracle and/or its affiliates. 33
  • 54.
    Users & Privileges Another important task of MySQL DBAs is to manage who's accessing the database. Copyright @ 2021 Oracle and/or its affiliates. 34
  • 55.
    Users & Privileges Another important task of MySQL DBAs is to manage who's accessing the database. Security is very important for MySQL and 8.0 provides many features related to user accounts. We also introduced more granular privileges. Copyright @ 2021 Oracle and/or its affiliates. 34
  • 56.
    ROLES new more secureauthentication method caching_sha2_password comments & json a ributes random password more options regarding old passwords: history reuse interval current password requirement lock time, ... Users & Privileges An other important task of MySQL DBAs is to manage who's accessing the database. Security is very important for MySQL and 8.0 provides many features related to user accounts. We also introduced more granular privileges. MySQL 8.0 introduced: Copyright @ 2021 Oracle and/or its affiliates. 34
  • 57.
    Extending MySQL Shell Forrepetitive or complex tasks, it's also possible to extend MySQL Shell with plugins. Copyright @ 2021 Oracle and/or its affiliates. 35
  • 58.
    Extending MySQL Shell Forrepetitive or complex tasks, it's also possible to extend MySQL Shell with plugins. User account management is very good candidate for plugins. Copyright @ 2021 Oracle and/or its affiliates. 35
  • 59.
    Extending MySQL Shell Forrepetitive or complex tasks, it's also possible to extend MySQL Shell with plugins. User account management is very good candidate for plugins. Some words on MySQL Shell Plugins: can be wri en in Python or JavaScript are loaded when starting MySQL Shell can use any local modules too Copyright @ 2021 Oracle and/or its affiliates. 35
  • 60.
    Extending MySQL Shell Forrepetitive or complex tasks, it's also possible to extend MySQL Shell with plugins. User account management is very good candidate for plugins. Some words on MySQL Shell Plugins: can be wri en in Python or JavaScript are loaded when starting MySQL Shell can use any local modules too You can nd multiple plugins to test and extend on: h ps://github.com/lefred/mysqlshell-plugins Copyright @ 2021 Oracle and/or its affiliates. 35
  • 61.
    Users & Privileges- plugins examples Dumping all users and grants: Copyright @ 2021 Oracle and/or its affiliates. 36
  • 62.
    Users & Privileges- plugins examples Copy users from a server to another: Copyright @ 2021 Oracle and/or its affiliates. 37
  • 63.
    Users & Privileges- plugins examples Create a user wizard Copyright @ 2021 Oracle and/or its affiliates. 38
  • 64.
    Users & Privileges- plugins examples Create a user wizard Copyright @ 2021 Oracle and/or its affiliates. 38
  • 65.
    Users & Privileges- plugins examples Authentication methods and passwords to expire: Copyright @ 2021 Oracle and/or its affiliates. 39
  • 66.
    Understand the Workload What'shappening ? Copyright @ 2021 Oracle and/or its affiliates. 40
  • 67.
    Observability in MySQL8.0 Information_Schema, Performance_Schema and sys schema are in constant evolution and many new metrics appeared in MySQL 8.0. Copyright @ 2021 Oracle and/or its affiliates. 41
  • 68.
    Observability in MySQL8.0 Information_Schema, Performance_Schema and sys schema are in constant evolution and many new metrics appeared in MySQL 8.0. Unless you deal with those schemes everyday, it might be complicated to easily nd the information you are looking for. MySQL Shell can again be helpful here. Copyright @ 2021 Oracle and/or its affiliates. 41
  • 69.
    Observability in MySQL8.0 Information_Schema, Performance_Schema and sys schema are in constant evolution and many new metrics appeared in MySQL 8.0. Unless you deal with those schemes everyday, it might be complicated to easily nd the information you are looking for. MySQL Shell can again be helpful here. There are already several plugins that provide useful information to be er understand the workload of the database. Copyright @ 2021 Oracle and/or its affiliates. 41
  • 70.
    Workload examples Copyright @2021 Oracle and/or its affiliates. 42
  • 71.
    Workload examples Copyright @2021 Oracle and/or its affiliates. 42
  • 72.
    Workload examples Copyright @2021 Oracle and/or its affiliates. 43
  • 73.
    Workload examples More infoand examples: h ps://github.com/lefred/mysqlshell-plugins/wiki Copyright @ 2021 Oracle and/or its affiliates. 43
  • 74.
    Fight the slowand ine cient queries ! JS > check.getFullTableScanQuery() *************************** 1. row *************************** schema_name: test sum_rows_examined: 21 avg_rows_call: 21.0000 tot_lat: 98.20 ms exec_count: 1 latency_per_call: 98.20 ms rst_seen: 2020-07-31 11:51:16.423903 last_seen: 2020-07-31 11:51:16.423903 query_sample_text: select * from users Copyright @ 2021 Oracle and/or its affiliates. 44
  • 75.
    Fight the slowand ine cient queries ! (2) Do you want to have EXPLAIN output? (y/N) y *************************** 1. row *************************** id: 1 select_type: SIMPLE table: users partitions: NULL type: ALL possible_keys: NULL key: NULL key_len: NULL ref: NULL rows: 18 ltered: 100 Extra: NULL Copyright @ 2021 Oracle and/or its affiliates. 45
  • 76.
    Fight the slowand ine cient queries ! (3) Do you want to have EXPLAIN in JSON format output? (y/N) y *************************** 1. row *************************** EXPLAIN: { "query_block": { "select_id": 1, "cost_info": { "query_cost": "2.80" }, "table": { "table_name": "users", "access_type": "ALL", "rows_examined_per_scan": 18, "rows_produced_per_join": 18, " ltered": "100.00", "cost_info": { "read_cost": "1.00", "eval_cost": "1.80", "pre x_cost": "2.80", "data_read_per_join": "576" }, "used_columns": [ "user_id", "user_name", "salary" ] } } } Copyright @ 2021 Oracle and/or its affiliates. 46
  • 77.
    Fight the slowand ine cient queries ! (4) Do you want to have EXPLAIN in TREE format output? (y/N) y *************************** 1. row *************************** EXPLAIN: -> Table scan on users (cost=2.80 rows=18) Do you want to have EXPLAIN ANALYZE output? (y/N) y *************************** 1. row *************************** EXPLAIN: -> Table scan on users (cost=2.80 rows=18) (actual time=0.041..0.067 rows=21 loops=1) Copyright @ 2021 Oracle and/or its affiliates. 47
  • 78.
    Do you rememberthe old SHOW PROFILE ? You can get similar information using the pro ling plugin for MySQL Shell MySQL Shell - Pro ler Copyright @ 2021 Oracle and/or its affiliates. 48
  • 79.
    And also foranother foreground thread: MySQL Shell - Pro ler (2) Copyright @ 2021 Oracle and/or its affiliates. 49
  • 80.
    Query Execution Plan Anothervery important part of the MySQL DBA is to control the QEP of the queries. There is a plugin that allows you to run the QEP for a query, save it and compare them. It supports di erent MySQL 8.0 versions. Copyright @ 2021 Oracle and/or its affiliates. 50
  • 81.
    Query Execution Plan(2) Copyright @ 2021 Oracle and/or its affiliates. 51
  • 82.
    Use and contributeto MySQL Shell Plugins Get the code from h ps://github.com/lefred/mysqlshell-plugins and Pull Requests are welcome ! Copyright @ 2021 Oracle and/or its affiliates. 52
  • 83.
    Something more... more control Copyright@ 2021 Oracle and/or its affiliates. 53
  • 84.
    con g.getGlobalNonDefault() con g.getSessionNonDefault() cong.getPersistedVariables() con g.getVariableInfo() Who changed what ? Sometimes the DBA needs to ght with con guration changes... This information related to variables is available from the con g MySQL Shell Plugin: Copyright @ 2021 Oracle and/or its affiliates. 54
  • 85.
    con g.getGlobalNonDefault() con g.getSessionNonDefault() cong.getPersistedVariables() con g.getVariableInfo() Who changed what ? Sometimes the DBA needs to ght with con guration changes... This information related to variables is available from the con g MySQL Shell Plugin: Copyright @ 2021 Oracle and/or its affiliates. 54
  • 86.
    Con guring Variables- MySQL Shell (2) Copyright @ 2021 Oracle and/or its affiliates. 55
  • 87.
    Con guring Variables- MySQL Shell (3) Copyright @ 2021 Oracle and/or its affiliates. 56
  • 88.
    Thank you ! Copyright@ 2021 Oracle and/or its affiliates. 57
  • 89.
    Questions ? Copyright @2021 Oracle and/or its affiliates. 58